Note: This is test shard 1 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/0, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)1> >
[ RUN ] AllTypesItest/0.TestAllKeyTypes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260811 02:59:16.840240 22243 test_util.cc:268] Using random seed: -1475486980
2026-08-11T02:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:59:16Z Disabled control of system clock
I20260811 02:59:16.883119 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.184.254:39103
--webserver_interface=127.21.184.254
--webserver_port=0
--builtin_ntp_servers=127.21.184.212:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.184.254:39103 with env {}
W20260811 02:59:17.234809 22252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:17.235167 22252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:17.235316 22252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:17.242902 22252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:59:17.243032 22252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:17.243119 22252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:59:17.243237 22252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:59:17.253002 22252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.184.254:39103
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.184.254:39103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.184.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:17.254534 22252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:17.256433 22252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:17.269362 22259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:17.269449 22261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:17.269730 22252 server_base.cc:1034] running on GCE node
W20260811 02:59:17.269435 22258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:17.270915 22252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:17.272424 22252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:17.273717 22252 hybrid_clock.cc:648] HybridClock initialized: now 1786417157273639 us; error 93 us; skew 500 ppm
I20260811 02:59:17.274044 22252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:17.276397 22252 webserver.cc:460] Webserver started at http://127.21.184.254:35513/ using document root <none> and password file <none>
I20260811 02:59:17.277096 22252 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:17.277204 22252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:17.277480 22252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:17.280346 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/instance:
uuid: "730573160f5549709951e9c5d87fd997"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.281155 22252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal/instance:
uuid: "730573160f5549709951e9c5d87fd997"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.286808 22252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260811 02:59:17.290699 22267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:17.292881 22252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260811 02:59:17.293136 22252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
uuid: "730573160f5549709951e9c5d87fd997"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.293452 22252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:17.316766 22252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:17.317950 22252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:17.318342 22252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:17.338277 22252 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.254:39103
I20260811 02:59:17.338289 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.254:39103 every 8 connection(s)
I20260811 02:59:17.339888 22252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
I20260811 02:59:17.344167 22324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:17.345407 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22252
I20260811 02:59:17.345752 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal/instance
I20260811 02:59:17.355603 22324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997: Bootstrap starting.
I20260811 02:59:17.358945 22324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:17.360055 22324 log.cc:824] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:17.363481 22324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997: No bootstrap required, opened a new log
I20260811 02:59:17.373898 22324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } }
I20260811 02:59:17.374336 22324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:17.374457 22324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730573160f5549709951e9c5d87fd997, State: Initialized, Role: FOLLOWER
I20260811 02:59:17.375178 22324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } }
I20260811 02:59:17.375492 22324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:59:17.375666 22324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:59:17.375888 22324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:17.378859 22324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } }
I20260811 02:59:17.379488 22324 leader_election.cc:302] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730573160f5549709951e9c5d87fd997; no voters:
I20260811 02:59:17.380189 22324 leader_election.cc:288] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:59:17.380357 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:17.381868 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [term 1 LEADER]: Becoming Leader. State: Replica: 730573160f5549709951e9c5d87fd997, State: Running, Role: LEADER
I20260811 02:59:17.382424 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } }
I20260811 02:59:17.382900 22324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:59:17.386018 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730573160f5549709951e9c5d87fd997. Latest consensus state: current_term: 1 leader_uuid: "730573160f5549709951e9c5d87fd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } } }
I20260811 02:59:17.386147 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730573160f5549709951e9c5d87fd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730573160f5549709951e9c5d87fd997" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 39103 } } }
I20260811 02:59:17.386610 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [sys.catalog]: This master's current role is: LEADER
I20260811 02:59:17.386620 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997 [sys.catalog]: This master's current role is: LEADER
I20260811 02:59:17.388517 22335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:59:17.394421 22335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:59:17.404665 22335 catalog_manager.cc:1347] Generated new cluster ID: fb70519437cd41dc87a177b9a94846c3
I20260811 02:59:17.404814 22335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:59:17.436882 22335 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:59:17.438387 22335 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:59:17.446435 22335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730573160f5549709951e9c5d87fd997: Generated new TSK 0
I20260811 02:59:17.447388 22335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:59:17.459249 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.193:0
--local_ip_for_outbound_sockets=127.21.184.193
--webserver_interface=127.21.184.193
--webserver_port=0
--tserver_master_addrs=127.21.184.254:39103
--builtin_ntp_servers=127.21.184.212:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260811 02:59:17.816321 22348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:17.816658 22348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:17.816823 22348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:17.824575 22348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:17.824767 22348 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:17.824865 22348 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:17.825037 22348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.193
I20260811 02:59:17.834684 22348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.184.193
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:39103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.193
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:17.836236 22348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:17.838332 22348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:17.853302 22353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:17.853251 22356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:17.853415 22358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:17.854264 22348 server_base.cc:1034] running on GCE node
I20260811 02:59:17.854987 22348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:17.856062 22348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:17.857342 22348 hybrid_clock.cc:648] HybridClock initialized: now 1786417157857262 us; error 34 us; skew 500 ppm
I20260811 02:59:17.857796 22348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:17.860541 22348 webserver.cc:460] Webserver started at http://127.21.184.193:46837/ using document root <none> and password file <none>
I20260811 02:59:17.861380 22348 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:17.861573 22348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:17.861984 22348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:17.865108 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/instance:
uuid: "e50fe0e1ec3e41e195850168afc88c59"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.866043 22348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal/instance:
uuid: "e50fe0e1ec3e41e195850168afc88c59"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.871929 22348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 02:59:17.875821 22365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:17.877316 22348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 02:59:17.877521 22348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
uuid: "e50fe0e1ec3e41e195850168afc88c59"
format_stamp: "Formatted at 2026-08-11 02:59:17 on dist-test-slave-p3t5"
I20260811 02:59:17.877821 22348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:17.900321 22348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:17.901297 22348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:17.901706 22348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:17.902973 22348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:17.904666 22348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:17.904767 22348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:17.904901 22348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:17.904963 22348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:17.941193 22348 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.193:45179
I20260811 02:59:17.941233 22491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.193:45179 every 8 connection(s)
I20260811 02:59:17.942623 22348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
I20260811 02:59:17.943375 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22348
I20260811 02:59:17.943626 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal/instance
I20260811 02:59:17.948318 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.194:0
--local_ip_for_outbound_sockets=127.21.184.194
--webserver_interface=127.21.184.194
--webserver_port=0
--tserver_master_addrs=127.21.184.254:39103
--builtin_ntp_servers=127.21.184.212:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 02:59:17.959501 22492 heartbeater.cc:347] Connected to a master server at 127.21.184.254:39103
I20260811 02:59:17.959887 22492 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:17.960750 22492 heartbeater.cc:511] Master 127.21.184.254:39103 requested a full tablet report, sending...
I20260811 02:59:17.963507 22286 ts_manager.cc:194] Registered new tserver with Master: e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179)
I20260811 02:59:17.966038 22286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.193:38879
W20260811 02:59:18.311262 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:18.311610 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:18.311712 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:18.319149 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:18.319307 22496 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:18.319348 22496 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:18.319430 22496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.194
I20260811 02:59:18.329206 22496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.184.194
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:39103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.194
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:18.330747 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:18.332721 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:18.348392 22504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:18.348421 22502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:18.349408 22506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:18.349730 22496 server_base.cc:1034] running on GCE node
I20260811 02:59:18.350519 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:18.351642 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:18.352918 22496 hybrid_clock.cc:648] HybridClock initialized: now 1786417158352847 us; error 50 us; skew 500 ppm
I20260811 02:59:18.353276 22496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:18.356102 22496 webserver.cc:460] Webserver started at http://127.21.184.194:40507/ using document root <none> and password file <none>
I20260811 02:59:18.356881 22496 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:18.357054 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:18.357414 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:18.360363 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/instance:
uuid: "22619620ec6945ceb0039c90abfd1315"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.361310 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal/instance:
uuid: "22619620ec6945ceb0039c90abfd1315"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.367255 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 02:59:18.371392 22512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.372892 22496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 02:59:18.373113 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
uuid: "22619620ec6945ceb0039c90abfd1315"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.373354 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:18.398298 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:18.399307 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:18.399704 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:18.400904 22496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:18.402643 22496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:18.402763 22496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.402935 22496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:18.403014 22496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.440640 22496 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.194:42133
I20260811 02:59:18.440690 22634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.194:42133 every 8 connection(s)
I20260811 02:59:18.442142 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
I20260811 02:59:18.442783 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22496
I20260811 02:59:18.443002 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal/instance
I20260811 02:59:18.447073 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.195:0
--local_ip_for_outbound_sockets=127.21.184.195
--webserver_interface=127.21.184.195
--webserver_port=0
--tserver_master_addrs=127.21.184.254:39103
--builtin_ntp_servers=127.21.184.212:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 02:59:18.459146 22635 heartbeater.cc:347] Connected to a master server at 127.21.184.254:39103
I20260811 02:59:18.459523 22635 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:18.460407 22635 heartbeater.cc:511] Master 127.21.184.254:39103 requested a full tablet report, sending...
I20260811 02:59:18.462111 22286 ts_manager.cc:194] Registered new tserver with Master: 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133)
I20260811 02:59:18.462972 22286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.194:50173
W20260811 02:59:18.805536 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:18.805902 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:18.806084 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:18.813709 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:18.813860 22638 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:18.813900 22638 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:18.814109 22638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.195
I20260811 02:59:18.823839 22638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.184.195
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:39103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.195
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:18.825418 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:18.828034 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:18.842964 22644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:18.843051 22645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:18.843001 22647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:18.844317 22638 server_base.cc:1034] running on GCE node
I20260811 02:59:18.844969 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:18.845979 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:18.847205 22638 hybrid_clock.cc:648] HybridClock initialized: now 1786417158847113 us; error 60 us; skew 500 ppm
I20260811 02:59:18.847482 22638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:18.850044 22638 webserver.cc:460] Webserver started at http://127.21.184.195:33387/ using document root <none> and password file <none>
I20260811 02:59:18.850740 22638 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:18.850909 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:18.851212 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:18.863790 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/instance:
uuid: "1cf73c055d924d7f9873b7404b126b05"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.864650 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal/instance:
uuid: "1cf73c055d924d7f9873b7404b126b05"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.870734 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 02:59:18.874823 22654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.876381 22638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 02:59:18.876539 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
uuid: "1cf73c055d924d7f9873b7404b126b05"
format_stamp: "Formatted at 2026-08-11 02:59:18 on dist-test-slave-p3t5"
I20260811 02:59:18.876730 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:18.897898 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:18.898898 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:18.899242 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:18.900394 22638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:18.902254 22638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:18.902364 22638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.902532 22638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:18.902609 22638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:18.938747 22638 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.195:33829
I20260811 02:59:18.938796 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.195:33829 every 8 connection(s)
I20260811 02:59:18.940223 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
I20260811 02:59:18.942548 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22638
I20260811 02:59:18.942724 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal/instance
I20260811 02:59:18.956362 22781 heartbeater.cc:347] Connected to a master server at 127.21.184.254:39103
I20260811 02:59:18.956732 22781 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:18.957585 22781 heartbeater.cc:511] Master 127.21.184.254:39103 requested a full tablet report, sending...
I20260811 02:59:18.959590 22286 ts_manager.cc:194] Registered new tserver with Master: 1cf73c055d924d7f9873b7404b126b05 (127.21.184.195:33829)
I20260811 02:59:18.960682 22243 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:59:18.960714 22286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.195:60477
I20260811 02:59:18.969504 22492 heartbeater.cc:503] Master 127.21.184.254:39103 was elected leader, sending a full tablet report...
I20260811 02:59:18.984853 22286 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57520:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000*\004\001\000\000\000T"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:59:18.993458 22286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:59:19.027014 22565 tablet_service.cc:1467] Processing CreateTablet for tablet 11f15911e28149bc9a9204be778bf03c (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION VALUES < 42
I20260811 02:59:19.030894 22419 tablet_service.cc:1467] Processing CreateTablet for tablet 11f15911e28149bc9a9204be778bf03c (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION VALUES < 42
I20260811 02:59:19.028520 22563 tablet_service.cc:1467] Processing CreateTablet for tablet 21a1cbbde34846dabe1e027ad01b81bf (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260811 02:59:19.032684 22419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f15911e28149bc9a9204be778bf03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.031981 22710 tablet_service.cc:1467] Processing CreateTablet for tablet 11f15911e28149bc9a9204be778bf03c (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION VALUES < 42
I20260811 02:59:19.027257 22564 tablet_service.cc:1467] Processing CreateTablet for tablet 89079ef5b2c941abac257c0157d87e84 (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260811 02:59:19.033205 22563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a1cbbde34846dabe1e027ad01b81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.033645 22564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89079ef5b2c941abac257c0157d87e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.032024 22708 tablet_service.cc:1467] Processing CreateTablet for tablet 21a1cbbde34846dabe1e027ad01b81bf (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260811 02:59:19.034999 22708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a1cbbde34846dabe1e027ad01b81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.035358 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f15911e28149bc9a9204be778bf03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.038709 22417 tablet_service.cc:1467] Processing CreateTablet for tablet 21a1cbbde34846dabe1e027ad01b81bf (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260811 02:59:19.038733 22418 tablet_service.cc:1467] Processing CreateTablet for tablet 89079ef5b2c941abac257c0157d87e84 (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260811 02:59:19.039338 22417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a1cbbde34846dabe1e027ad01b81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.039615 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89079ef5b2c941abac257c0157d87e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.032128 22709 tablet_service.cc:1467] Processing CreateTablet for tablet 89079ef5b2c941abac257c0157d87e84 (DEFAULT_TABLE table=all-types-table [id=8a891bf5832645c6b88fc205c8518259]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260811 02:59:19.042248 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89079ef5b2c941abac257c0157d87e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.046119 22565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f15911e28149bc9a9204be778bf03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:19.058233 22799 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: Bootstrap starting.
I20260811 02:59:19.063218 22802 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: Bootstrap starting.
I20260811 02:59:19.069566 22799 tablet_bootstrap.cc:654] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.070307 22802 tablet_bootstrap.cc:654] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.071134 22799 log.cc:824] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:19.071636 22802 log.cc:824] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:19.072966 22801 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: Bootstrap starting.
I20260811 02:59:19.076717 22799 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: No bootstrap required, opened a new log
I20260811 02:59:19.077059 22799 ts_tablet_manager.cc:1397] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20260811 02:59:19.077083 22801 tablet_bootstrap.cc:654] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.079263 22801 log.cc:824] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:19.081198 22802 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: No bootstrap required, opened a new log
I20260811 02:59:19.081499 22802 ts_tablet_manager.cc:1397] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260811 02:59:19.082615 22801 tablet_bootstrap.cc:492] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: No bootstrap required, opened a new log
I20260811 02:59:19.082971 22801 ts_tablet_manager.cc:1397] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260811 02:59:19.088681 22799 raft_consensus.cc:348] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.089248 22799 raft_consensus.cc:374] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.089361 22799 raft_consensus.cc:729] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf73c055d924d7f9873b7404b126b05, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.089382 22802 raft_consensus.cc:348] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.089918 22802 raft_consensus.cc:374] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.090060 22802 raft_consensus.cc:729] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50fe0e1ec3e41e195850168afc88c59, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.090075 22799 consensus_queue.cc:260] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.090703 22802 consensus_queue.cc:260] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.094441 22802 ts_tablet_manager.cc:1428] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260811 02:59:19.094901 22802 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59: Bootstrap starting.
I20260811 02:59:19.095357 22801 raft_consensus.cc:348] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.096015 22801 raft_consensus.cc:374] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.096158 22801 raft_consensus.cc:729] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22619620ec6945ceb0039c90abfd1315, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.096776 22802 tablet_bootstrap.cc:654] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.097079 22801 consensus_queue.cc:260] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.098505 22781 heartbeater.cc:503] Master 127.21.184.254:39103 was elected leader, sending a full tablet report...
I20260811 02:59:19.099432 22799 ts_tablet_manager.cc:1428] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05: Time spent starting tablet: real 0.022s user 0.019s sys 0.006s
I20260811 02:59:19.099941 22799 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: Bootstrap starting.
I20260811 02:59:19.103741 22802 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59: No bootstrap required, opened a new log
I20260811 02:59:19.104018 22802 ts_tablet_manager.cc:1397] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 02:59:19.104372 22799 tablet_bootstrap.cc:654] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.105261 22801 ts_tablet_manager.cc:1428] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315: Time spent starting tablet: real 0.022s user 0.009s sys 0.009s
I20260811 02:59:19.105217 22802 raft_consensus.cc:348] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.105511 22802 raft_consensus.cc:374] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.105695 22802 raft_consensus.cc:729] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50fe0e1ec3e41e195850168afc88c59, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.105844 22635 heartbeater.cc:503] Master 127.21.184.254:39103 was elected leader, sending a full tablet report...
I20260811 02:59:19.106048 22801 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: Bootstrap starting.
I20260811 02:59:19.106081 22802 consensus_queue.cc:260] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.107597 22802 ts_tablet_manager.cc:1428] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:59:19.107877 22802 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: Bootstrap starting.
I20260811 02:59:19.108518 22801 tablet_bootstrap.cc:654] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.109825 22802 tablet_bootstrap.cc:654] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.113272 22799 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: No bootstrap required, opened a new log
I20260811 02:59:19.113512 22799 ts_tablet_manager.cc:1397] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20260811 02:59:19.114889 22799 raft_consensus.cc:348] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.115237 22799 raft_consensus.cc:374] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.115370 22799 raft_consensus.cc:729] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf73c055d924d7f9873b7404b126b05, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.115741 22799 consensus_queue.cc:260] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.116377 22799 ts_tablet_manager.cc:1428] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:59:19.116659 22799 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05: Bootstrap starting.
I20260811 02:59:19.118523 22799 tablet_bootstrap.cc:654] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.120641 22801 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: No bootstrap required, opened a new log
I20260811 02:59:19.120841 22801 ts_tablet_manager.cc:1397] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260811 02:59:19.121026 22799 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05: No bootstrap required, opened a new log
I20260811 02:59:19.121194 22799 ts_tablet_manager.cc:1397] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 02:59:19.121938 22799 raft_consensus.cc:348] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.121963 22801 raft_consensus.cc:348] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.122187 22801 raft_consensus.cc:374] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.122205 22799 raft_consensus.cc:374] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.122280 22799 raft_consensus.cc:729] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf73c055d924d7f9873b7404b126b05, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.122355 22801 raft_consensus.cc:729] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22619620ec6945ceb0039c90abfd1315, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.122589 22799 consensus_queue.cc:260] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.122705 22801 consensus_queue.cc:260] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.123301 22802 tablet_bootstrap.cc:492] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: No bootstrap required, opened a new log
I20260811 02:59:19.123281 22799 ts_tablet_manager.cc:1428] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:59:19.123481 22802 ts_tablet_manager.cc:1397] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260811 02:59:19.123921 22801 ts_tablet_manager.cc:1428] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:59:19.124193 22801 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315: Bootstrap starting.
I20260811 02:59:19.124290 22802 raft_consensus.cc:348] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.124548 22802 raft_consensus.cc:374] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.124624 22802 raft_consensus.cc:729] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50fe0e1ec3e41e195850168afc88c59, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.124903 22802 consensus_queue.cc:260] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.125964 22801 tablet_bootstrap.cc:654] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:19.126262 22802 ts_tablet_manager.cc:1428] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260811 02:59:19.130009 22801 tablet_bootstrap.cc:492] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315: No bootstrap required, opened a new log
I20260811 02:59:19.130155 22801 ts_tablet_manager.cc:1397] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260811 02:59:19.130726 22801 raft_consensus.cc:348] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.130915 22801 raft_consensus.cc:374] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:19.130967 22801 raft_consensus.cc:729] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22619620ec6945ceb0039c90abfd1315, State: Initialized, Role: FOLLOWER
I20260811 02:59:19.131209 22801 consensus_queue.cc:260] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.131644 22801 ts_tablet_manager.cc:1428] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 02:59:19.174590 22806 raft_consensus.cc:482] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:19.175060 22806 raft_consensus.cc:504] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.177035 22806 leader_election.cc:288] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179), 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133)
I20260811 02:59:19.184863 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89079ef5b2c941abac257c0157d87e84" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50fe0e1ec3e41e195850168afc88c59" is_pre_election: true
I20260811 02:59:19.185528 22440 raft_consensus.cc:2454] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 0.
I20260811 02:59:19.185660 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89079ef5b2c941abac257c0157d87e84" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315" is_pre_election: true
I20260811 02:59:19.186260 22585 raft_consensus.cc:2454] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 0.
I20260811 02:59:19.186301 22656 leader_election.cc:302] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf73c055d924d7f9873b7404b126b05, e50fe0e1ec3e41e195850168afc88c59; no voters:
I20260811 02:59:19.186801 22806 raft_consensus.cc:2781] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:19.186976 22806 raft_consensus.cc:482] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:19.187112 22806 raft_consensus.cc:3037] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.189656 22806 raft_consensus.cc:504] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.190407 22806 leader_election.cc:288] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 election: Requested vote from peers e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179), 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133)
I20260811 02:59:19.190917 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89079ef5b2c941abac257c0157d87e84" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50fe0e1ec3e41e195850168afc88c59"
I20260811 02:59:19.191026 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89079ef5b2c941abac257c0157d87e84" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315"
I20260811 02:59:19.191144 22440 raft_consensus.cc:3037] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.191283 22585 raft_consensus.cc:3037] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Advancing to term 1
W20260811 02:59:19.192109 22782 tablet.cc:2369] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:59:19.193729 22585 raft_consensus.cc:2454] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 1.
I20260811 02:59:19.194026 22440 raft_consensus.cc:2454] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 1.
I20260811 02:59:19.194218 22658 leader_election.cc:302] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf73c055d924d7f9873b7404b126b05, 22619620ec6945ceb0039c90abfd1315; no voters:
W20260811 02:59:19.194572 22636 tablet.cc:2369] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:59:19.194626 22806 raft_consensus.cc:2781] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:19.195133 22806 raft_consensus.cc:686] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf73c055d924d7f9873b7404b126b05, State: Running, Role: LEADER
W20260811 02:59:19.195438 22493 tablet.cc:2369] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:59:19.196092 22806 consensus_queue.cc:237] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.201220 22285 catalog_manager.cc:5495] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf73c055d924d7f9873b7404b126b05 (127.21.184.195). New cstate: current_term: 1 leader_uuid: "1cf73c055d924d7f9873b7404b126b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } health_report { overall_health: HEALTHY } } }
I20260811 02:59:19.209059 22808 raft_consensus.cc:482] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:19.209321 22808 raft_consensus.cc:504] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.211041 22808 leader_election.cc:288] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133), 1cf73c055d924d7f9873b7404b126b05 (127.21.184.195:33829)
I20260811 02:59:19.219254 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a1cbbde34846dabe1e027ad01b81bf" candidate_uuid: "e50fe0e1ec3e41e195850168afc88c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315" is_pre_election: true
I20260811 02:59:19.219558 22585 raft_consensus.cc:2454] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50fe0e1ec3e41e195850168afc88c59 in term 0.
I20260811 02:59:19.219535 22731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a1cbbde34846dabe1e027ad01b81bf" candidate_uuid: "e50fe0e1ec3e41e195850168afc88c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf73c055d924d7f9873b7404b126b05" is_pre_election: true
I20260811 02:59:19.220026 22731 raft_consensus.cc:2454] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50fe0e1ec3e41e195850168afc88c59 in term 0.
I20260811 02:59:19.220063 22369 leader_election.cc:302] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22619620ec6945ceb0039c90abfd1315, e50fe0e1ec3e41e195850168afc88c59; no voters:
I20260811 02:59:19.220463 22808 raft_consensus.cc:2781] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:19.220597 22808 raft_consensus.cc:482] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:19.220712 22808 raft_consensus.cc:3037] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.222868 22808 raft_consensus.cc:504] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.223589 22808 leader_election.cc:288] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [CANDIDATE]: Term 1 election: Requested vote from peers 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133), 1cf73c055d924d7f9873b7404b126b05 (127.21.184.195:33829)
I20260811 02:59:19.224030 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a1cbbde34846dabe1e027ad01b81bf" candidate_uuid: "e50fe0e1ec3e41e195850168afc88c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315"
I20260811 02:59:19.224193 22731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a1cbbde34846dabe1e027ad01b81bf" candidate_uuid: "e50fe0e1ec3e41e195850168afc88c59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf73c055d924d7f9873b7404b126b05"
I20260811 02:59:19.224303 22585 raft_consensus.cc:3037] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.224428 22731 raft_consensus.cc:3037] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.227360 22585 raft_consensus.cc:2454] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50fe0e1ec3e41e195850168afc88c59 in term 1.
I20260811 02:59:19.227451 22731 raft_consensus.cc:2454] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50fe0e1ec3e41e195850168afc88c59 in term 1.
I20260811 02:59:19.227969 22367 leader_election.cc:302] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cf73c055d924d7f9873b7404b126b05, 22619620ec6945ceb0039c90abfd1315, e50fe0e1ec3e41e195850168afc88c59; no voters:
I20260811 02:59:19.228374 22808 raft_consensus.cc:2781] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:19.228828 22808 raft_consensus.cc:686] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [term 1 LEADER]: Becoming Leader. State: Replica: e50fe0e1ec3e41e195850168afc88c59, State: Running, Role: LEADER
I20260811 02:59:19.229364 22808 consensus_queue.cc:237] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } }
I20260811 02:59:19.234601 22285 catalog_manager.cc:5495] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193). New cstate: current_term: 1 leader_uuid: "e50fe0e1ec3e41e195850168afc88c59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } health_report { overall_health: UNKNOWN } } }
I20260811 02:59:19.421993 22806 raft_consensus.cc:482] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:19.422315 22806 raft_consensus.cc:504] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.423101 22806 leader_election.cc:288] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133), e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179)
I20260811 02:59:19.423797 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f15911e28149bc9a9204be778bf03c" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50fe0e1ec3e41e195850168afc88c59" is_pre_election: true
I20260811 02:59:19.423902 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f15911e28149bc9a9204be778bf03c" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315" is_pre_election: true
I20260811 02:59:19.424100 22440 raft_consensus.cc:2454] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 0.
I20260811 02:59:19.424296 22585 raft_consensus.cc:2454] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 0.
I20260811 02:59:19.424674 22656 leader_election.cc:302] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf73c055d924d7f9873b7404b126b05, e50fe0e1ec3e41e195850168afc88c59; no voters:
I20260811 02:59:19.425024 22806 raft_consensus.cc:2781] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:19.425127 22806 raft_consensus.cc:482] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:19.425294 22806 raft_consensus.cc:3037] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.428154 22806 raft_consensus.cc:504] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.428869 22806 leader_election.cc:288] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 election: Requested vote from peers 22619620ec6945ceb0039c90abfd1315 (127.21.184.194:42133), e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179)
I20260811 02:59:19.429538 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f15911e28149bc9a9204be778bf03c" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50fe0e1ec3e41e195850168afc88c59"
I20260811 02:59:19.429557 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f15911e28149bc9a9204be778bf03c" candidate_uuid: "1cf73c055d924d7f9873b7404b126b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22619620ec6945ceb0039c90abfd1315"
I20260811 02:59:19.429890 22440 raft_consensus.cc:3037] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.429920 22585 raft_consensus.cc:3037] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:19.432125 22585 raft_consensus.cc:2454] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 1.
I20260811 02:59:19.432132 22440 raft_consensus.cc:2454] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf73c055d924d7f9873b7404b126b05 in term 1.
I20260811 02:59:19.432705 22658 leader_election.cc:302] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cf73c055d924d7f9873b7404b126b05, 22619620ec6945ceb0039c90abfd1315, e50fe0e1ec3e41e195850168afc88c59; no voters:
I20260811 02:59:19.433089 22806 raft_consensus.cc:2781] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:19.433307 22806 raft_consensus.cc:686] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf73c055d924d7f9873b7404b126b05, State: Running, Role: LEADER
I20260811 02:59:19.433601 22806 consensus_queue.cc:237] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } }
I20260811 02:59:19.436582 22285 catalog_manager.cc:5495] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf73c055d924d7f9873b7404b126b05 (127.21.184.195). New cstate: current_term: 1 leader_uuid: "1cf73c055d924d7f9873b7404b126b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 } health_report { overall_health: UNKNOWN } } }
I20260811 02:59:19.528455 22731 raft_consensus.cc:1261] T 21a1cbbde34846dabe1e027ad01b81bf P 1cf73c055d924d7f9873b7404b126b05 [term 1 FOLLOWER]: Refusing update from remote peer e50fe0e1ec3e41e195850168afc88c59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.529402 22440 raft_consensus.cc:1261] T 11f15911e28149bc9a9204be778bf03c P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Refusing update from remote peer 1cf73c055d924d7f9873b7404b126b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.529450 22584 raft_consensus.cc:1261] T 11f15911e28149bc9a9204be778bf03c P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Refusing update from remote peer 1cf73c055d924d7f9873b7404b126b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.529542 22808 consensus_queue.cc:1035] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf73c055d924d7f9873b7404b126b05" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.530470 22818 consensus_queue.cc:1035] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.529450 22585 raft_consensus.cc:1261] T 21a1cbbde34846dabe1e027ad01b81bf P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Refusing update from remote peer e50fe0e1ec3e41e195850168afc88c59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.531610 22808 consensus_queue.cc:1035] T 21a1cbbde34846dabe1e027ad01b81bf P e50fe0e1ec3e41e195850168afc88c59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.533012 22806 consensus_queue.cc:1035] T 11f15911e28149bc9a9204be778bf03c P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.539590 22584 raft_consensus.cc:1261] T 89079ef5b2c941abac257c0157d87e84 P 22619620ec6945ceb0039c90abfd1315 [term 1 FOLLOWER]: Refusing update from remote peer 1cf73c055d924d7f9873b7404b126b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.540983 22806 consensus_queue.cc:1035] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22619620ec6945ceb0039c90abfd1315" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.558730 22831 mvcc.cc:205] Tried to move back new op lower bound from 7317164685436170240 to 7317164684071407616. Current Snapshot: MvccSnapshot[applied={T|T < 7317164685436170240}]
I20260811 02:59:19.559533 22440 raft_consensus.cc:1261] T 89079ef5b2c941abac257c0157d87e84 P e50fe0e1ec3e41e195850168afc88c59 [term 1 FOLLOWER]: Refusing update from remote peer 1cf73c055d924d7f9873b7404b126b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:59:19.560967 22833 consensus_queue.cc:1035] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50fe0e1ec3e41e195850168afc88c59" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:19.568574 22839 mvcc.cc:205] Tried to move back new op lower bound from 7317164685415550976 to 7317164685042933760. Current Snapshot: MvccSnapshot[applied={T|T < 7317164685415550976}]
I20260811 02:59:19.573704 22829 mvcc.cc:205] Tried to move back new op lower bound from 7317164685415530496 to 7317164684207718400. Current Snapshot: MvccSnapshot[applied={T|T < 7317164685415530496}]
I20260811 02:59:19.717444 22564 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 02:59:19.717549 22708 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:19.718052 22419 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
730573160f5549709951e9c5d87fd997 | 127.21.184.254:39103 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:37011 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
1cf73c055d924d7f9873b7404b126b05 | 127.21.184.195:33829 | HEALTHY | <none> | 2 | 0
22619620ec6945ceb0039c90abfd1315 | 127.21.184.194:42133 | HEALTHY | <none> | 0 | 0
e50fe0e1ec3e41e195850168afc88c59 | 127.21.184.193:45179 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 1 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:45179
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:42133
local_ip_for_outbound_sockets | 127.21.184.195 | experimental | 127.21.184.195:33829
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:45179
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:42133
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_0.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb | hidden | 127.21.184.195:33829
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:37011 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+---------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260811 02:59:19.771088 22243 log_verifier.cc:126] Checking tablet 11f15911e28149bc9a9204be778bf03c
I20260811 02:59:19.802901 22243 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 11f15911e28149bc9a9204be778bf03c
I20260811 02:59:19.803079 22243 log_verifier.cc:126] Checking tablet 21a1cbbde34846dabe1e027ad01b81bf
I20260811 02:59:19.831179 22243 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 21a1cbbde34846dabe1e027ad01b81bf
I20260811 02:59:19.831292 22243 log_verifier.cc:126] Checking tablet 89079ef5b2c941abac257c0157d87e84
I20260811 02:59:19.859058 22243 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 89079ef5b2c941abac257c0157d87e84
I20260811 02:59:20.047851 22243 all_types-itest.cc:600] Scanning tablet: [, int8 key=42)
I20260811 02:59:20.178138 22243 all_types-itest.cc:600] Scanning tablet: [int8 key=42, int8 key=84)
I20260811 02:59:20.207854 22243 all_types-itest.cc:600] Scanning tablet: [int8 key=84, )
I20260811 02:59:20.222746 22243 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8YxRLs/build/asan/bin/kudu pid 22348
I20260811 02:59:20.248088 22486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 02:59:20.373920 22243 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8YxRLs/build/asan/bin/kudu with pid 22348
I20260811 02:59:20.398687 22243 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8YxRLs/build/asan/bin/kudu pid 22496
I20260811 02:59:20.406291 22629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 02:59:20.550529 22243 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8YxRLs/build/asan/bin/kudu with pid 22496
W20260811 02:59:20.580588 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.193:45179: connect: Connection refused (error 111)
I20260811 02:59:20.581746 22243 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8YxRLs/build/asan/bin/kudu pid 22638
W20260811 02:59:20.582868 22656 consensus_peers.cc:487] T 89079ef5b2c941abac257c0157d87e84 P 1cf73c055d924d7f9873b7404b126b05 -> Peer e50fe0e1ec3e41e195850168afc88c59 (127.21.184.193:45179): Couldn't send request to peer e50fe0e1ec3e41e195850168afc88c59. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.193:45179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 02:59:20.589205 22776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 02:59:20.689745 22243 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8YxRLs/build/asan/bin/kudu with pid 22638
I20260811 02:59:20.708967 22243 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8YxRLs/build/asan/bin/kudu pid 22252
I20260811 02:59:20.710194 22319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 02:59:20.823314 22243 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8YxRLs/build/asan/bin/kudu with pid 22252
2026-08-11T02:59:20Z chronyd exiting
[ OK ] AllTypesItest/0.TestAllKeyTypes (4040 ms)
[----------] 1 test from AllTypesItest/0 (4041 ms total)
[----------] 1 test from AllTypesItest/4, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> >
[ RUN ] AllTypesItest/4.TestAllKeyTypes
I20260811 02:59:20.879534 22243 test_util.cc:268] Using random seed: -1471447676
2026-08-11T02:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:59:20Z Disabled control of system clock
I20260811 02:59:20.913780 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.184.254:33799
--webserver_interface=127.21.184.254
--webserver_port=0
--builtin_ntp_servers=127.21.184.212:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.184.254:33799 with env {}
W20260811 02:59:21.266095 22895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:21.266428 22895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:21.266594 22895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:21.274271 22895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:59:21.274398 22895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:21.274475 22895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:59:21.274580 22895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:59:21.284243 22895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.184.254:33799
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.184.254:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.184.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:21.285912 22895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:21.287901 22895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:21.301093 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:21.300976 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:21.301023 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:21.301939 22895 server_base.cc:1034] running on GCE node
I20260811 02:59:21.302937 22895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:21.304530 22895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:21.305749 22895 hybrid_clock.cc:648] HybridClock initialized: now 1786417161305677 us; error 57 us; skew 500 ppm
I20260811 02:59:21.306157 22895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:21.312285 22895 webserver.cc:460] Webserver started at http://127.21.184.254:44645/ using document root <none> and password file <none>
I20260811 02:59:21.313129 22895 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:21.313313 22895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:21.313733 22895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:21.317162 22895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/instance:
uuid: "37817f3438924819a8b313e319f2be3c"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.318097 22895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal/instance:
uuid: "37817f3438924819a8b313e319f2be3c"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.324079 22895 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260811 02:59:21.328890 22910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:21.330569 22895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 02:59:21.330798 22895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
uuid: "37817f3438924819a8b313e319f2be3c"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.331063 22895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:21.348974 22895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:21.350070 22895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:21.350436 22895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:21.370728 22895 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.254:33799
I20260811 02:59:21.370760 22968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.254:33799 every 8 connection(s)
I20260811 02:59:21.372339 22895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb
I20260811 02:59:21.376443 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:21.377394 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22895
I20260811 02:59:21.377645 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/wal/instance
I20260811 02:59:21.390396 22969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: Bootstrap starting.
I20260811 02:59:21.394392 22969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:21.395761 22969 log.cc:824] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:21.399009 22969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: No bootstrap required, opened a new log
I20260811 02:59:21.406817 22969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } }
I20260811 02:59:21.407275 22969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:21.407405 22969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37817f3438924819a8b313e319f2be3c, State: Initialized, Role: FOLLOWER
I20260811 02:59:21.408136 22969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } }
I20260811 02:59:21.408432 22969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:59:21.408593 22969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:59:21.408782 22969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:21.411434 22969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } }
I20260811 02:59:21.412067 22969 leader_election.cc:302] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37817f3438924819a8b313e319f2be3c; no voters:
I20260811 02:59:21.412693 22969 leader_election.cc:288] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:59:21.412873 22974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:21.413688 22974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [term 1 LEADER]: Becoming Leader. State: Replica: 37817f3438924819a8b313e319f2be3c, State: Running, Role: LEADER
I20260811 02:59:21.414319 22974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } }
I20260811 02:59:21.415385 22969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:59:21.417429 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37817f3438924819a8b313e319f2be3c. Latest consensus state: current_term: 1 leader_uuid: "37817f3438924819a8b313e319f2be3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } } }
I20260811 02:59:21.417331 22976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37817f3438924819a8b313e319f2be3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37817f3438924819a8b313e319f2be3c" member_type: VOTER last_known_addr { host: "127.21.184.254" port: 33799 } } }
I20260811 02:59:21.417761 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [sys.catalog]: This master's current role is: LEADER
I20260811 02:59:21.417758 22976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c [sys.catalog]: This master's current role is: LEADER
I20260811 02:59:21.418776 22980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:59:21.426190 22980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:59:21.438208 22980 catalog_manager.cc:1347] Generated new cluster ID: e5bcd3446e8241619d9e161e22525af2
I20260811 02:59:21.438359 22980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:59:21.504998 22980 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:59:21.506542 22980 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:59:21.516460 22980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: Generated new TSK 0
I20260811 02:59:21.517503 22980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:59:21.533919 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.193:0
--local_ip_for_outbound_sockets=127.21.184.193
--webserver_interface=127.21.184.193
--webserver_port=0
--tserver_master_addrs=127.21.184.254:33799
--builtin_ntp_servers=127.21.184.212:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260811 02:59:21.887555 22996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:21.887898 22996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:21.888072 22996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:21.895681 22996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:21.895864 22996 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:21.895967 22996 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:21.896136 22996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.193
I20260811 02:59:21.905843 22996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.184.193
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.193
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:21.907322 22996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:21.909297 22996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:21.923619 23001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:21.923601 23002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:21.923568 23004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:21.929833 22996 server_base.cc:1034] running on GCE node
I20260811 02:59:21.930564 22996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:21.931682 22996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:21.932952 22996 hybrid_clock.cc:648] HybridClock initialized: now 1786417161932879 us; error 70 us; skew 500 ppm
I20260811 02:59:21.933290 22996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:21.935956 22996 webserver.cc:460] Webserver started at http://127.21.184.193:46855/ using document root <none> and password file <none>
I20260811 02:59:21.936676 22996 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:21.936842 22996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:21.937188 22996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:21.940060 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/instance:
uuid: "f4c17323c7214a8d8dbb4f2e000a05c8"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.940894 22996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal/instance:
uuid: "f4c17323c7214a8d8dbb4f2e000a05c8"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.946688 22996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260811 02:59:21.950510 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:21.952062 22996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 02:59:21.952275 22996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
uuid: "f4c17323c7214a8d8dbb4f2e000a05c8"
format_stamp: "Formatted at 2026-08-11 02:59:21 on dist-test-slave-p3t5"
I20260811 02:59:21.952508 22996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:21.970671 22996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:21.971621 22996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:21.971962 22996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:21.972993 22996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:21.974668 22996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:21.974786 22996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:21.974947 22996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:21.975023 22996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.010265 22996 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.193:43161
I20260811 02:59:22.010306 23135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.193:43161 every 8 connection(s)
I20260811 02:59:22.011766 22996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb
I20260811 02:59:22.018083 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 22996
I20260811 02:59:22.018332 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/wal/instance
I20260811 02:59:22.022630 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.194:0
--local_ip_for_outbound_sockets=127.21.184.194
--webserver_interface=127.21.184.194
--webserver_port=0
--tserver_master_addrs=127.21.184.254:33799
--builtin_ntp_servers=127.21.184.212:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 02:59:22.031562 23137 heartbeater.cc:347] Connected to a master server at 127.21.184.254:33799
I20260811 02:59:22.032049 23137 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:22.033011 23137 heartbeater.cc:511] Master 127.21.184.254:33799 requested a full tablet report, sending...
I20260811 02:59:22.035516 22931 ts_manager.cc:194] Registered new tserver with Master: f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161)
I20260811 02:59:22.037155 22931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.193:50733
W20260811 02:59:22.392413 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:22.392848 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:22.393010 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:22.401678 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:22.401852 23142 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:22.401909 23142 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:22.401996 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.194
I20260811 02:59:22.411759 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.184.194
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.194
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:22.413167 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:22.415153 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:22.430025 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:22.430176 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:22.430051 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:22.431097 23142 server_base.cc:1034] running on GCE node
I20260811 02:59:22.431810 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:22.432906 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:22.434144 23142 hybrid_clock.cc:648] HybridClock initialized: now 1786417162434058 us; error 63 us; skew 500 ppm
I20260811 02:59:22.434480 23142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:22.437167 23142 webserver.cc:460] Webserver started at http://127.21.184.194:38881/ using document root <none> and password file <none>
I20260811 02:59:22.438017 23142 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:22.438134 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:22.438409 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:22.441118 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/instance:
uuid: "5264f57db51340f9900ee8ceb7998983"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.442000 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal/instance:
uuid: "5264f57db51340f9900ee8ceb7998983"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.447791 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 02:59:22.452342 23156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.454037 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 02:59:22.454262 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
uuid: "5264f57db51340f9900ee8ceb7998983"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.454528 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:22.474018 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:22.474948 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:22.475277 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:22.476380 23142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:22.478222 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:22.478325 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.478478 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:22.478559 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.517421 23142 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.194:45825
I20260811 02:59:22.517491 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.194:45825 every 8 connection(s)
I20260811 02:59:22.519084 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb
I20260811 02:59:22.529484 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 23142
I20260811 02:59:22.529706 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/wal/instance
I20260811 02:59:22.533776 22243 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8YxRLs/build/asan/bin/kudu
/tmp/dist-test-task8YxRLs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.184.195:0
--local_ip_for_outbound_sockets=127.21.184.195
--webserver_interface=127.21.184.195
--webserver_port=0
--tserver_master_addrs=127.21.184.254:33799
--builtin_ntp_servers=127.21.184.212:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 02:59:22.536697 23281 heartbeater.cc:347] Connected to a master server at 127.21.184.254:33799
I20260811 02:59:22.537114 23281 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:22.538268 23281 heartbeater.cc:511] Master 127.21.184.254:33799 requested a full tablet report, sending...
I20260811 02:59:22.540097 22931 ts_manager.cc:194] Registered new tserver with Master: 5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825)
I20260811 02:59:22.541087 22931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.194:59661
W20260811 02:59:22.895812 23284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:59:22.896171 23284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:59:22.896266 23284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:59:22.903955 23284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:59:22.904106 23284 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 02:59:22.904146 23284 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 02:59:22.904240 23284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.184.195
I20260811 02:59:22.913796 23284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.184.212:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.184.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.184.195
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.21.184.254:33799
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.184.195
--log_dir=/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 02:59:22.915346 23284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:59:22.917326 23284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:59:22.932024 23290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:22.932024 23291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:59:22.932740 23293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:59:22.933024 23284 server_base.cc:1034] running on GCE node
I20260811 02:59:22.933825 23284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:59:22.934950 23284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:59:22.936237 23284 hybrid_clock.cc:648] HybridClock initialized: now 1786417162936132 us; error 74 us; skew 500 ppm
I20260811 02:59:22.936614 23284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 02:59:22.939117 23284 webserver.cc:460] Webserver started at http://127.21.184.195:44721/ using document root <none> and password file <none>
I20260811 02:59:22.939882 23284 fs_manager.cc:359] Metadata directory not provided
I20260811 02:59:22.940047 23284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:59:22.940405 23284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:59:22.943356 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/instance:
uuid: "4cbab65af5824618b363b5a8b389be33"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.944258 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal/instance:
uuid: "4cbab65af5824618b363b5a8b389be33"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.950037 23284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 02:59:22.953801 23301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.955264 23284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 02:59:22.955466 23284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data,/tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
uuid: "4cbab65af5824618b363b5a8b389be33"
format_stamp: "Formatted at 2026-08-11 02:59:22 on dist-test-slave-p3t5"
I20260811 02:59:22.955693 23284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:59:22.981033 23284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:59:22.982040 23284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:59:22.982393 23284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:59:22.983500 23284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:59:22.985112 23284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:59:22.985227 23284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:22.985391 23284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:59:22.985473 23284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:59:23.024860 23284 rpc_server.cc:307] RPC server started. Bound to: 127.21.184.195:35255
I20260811 02:59:23.024915 23423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.184.195:35255 every 8 connection(s)
I20260811 02:59:23.026472 23284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/info.pb
I20260811 02:59:23.031231 22243 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8YxRLs/build/asan/bin/kudu as pid 23284
I20260811 02:59:23.031443 22243 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/wal/instance
I20260811 02:59:23.040659 23137 heartbeater.cc:503] Master 127.21.184.254:33799 was elected leader, sending a full tablet report...
I20260811 02:59:23.043509 23424 heartbeater.cc:347] Connected to a master server at 127.21.184.254:33799
I20260811 02:59:23.043829 23424 heartbeater.cc:464] Registering TS with master...
I20260811 02:59:23.044595 23424 heartbeater.cc:511] Master 127.21.184.254:33799 requested a full tablet report, sending...
I20260811 02:59:23.046254 22931 ts_manager.cc:194] Registered new tserver with Master: 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255)
I20260811 02:59:23.047242 22931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.184.195:49589
I20260811 02:59:23.048157 22243 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:59:23.064584 22931 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40486:
name: "all-types-table"
schema {
columns {
name: "key"
type: DECIMAL32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:59:23.072587 22931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:59:23.108359 23060 tablet_service.cc:1467] Processing CreateTablet for tablet 7c377df55c8049f6bd13fe8f53f52b28 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260811 02:59:23.108946 23061 tablet_service.cc:1467] Processing CreateTablet for tablet fb6662cff60c4865acd527a45a257580 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260811 02:59:23.108204 23062 tablet_service.cc:1467] Processing CreateTablet for tablet bbc3e9de57f1419aa78b468e7d684a89 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260811 02:59:23.108866 23351 tablet_service.cc:1467] Processing CreateTablet for tablet bbc3e9de57f1419aa78b468e7d684a89 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260811 02:59:23.108882 23349 tablet_service.cc:1467] Processing CreateTablet for tablet 7c377df55c8049f6bd13fe8f53f52b28 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260811 02:59:23.110612 23061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6662cff60c4865acd527a45a257580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.110870 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc3e9de57f1419aa78b468e7d684a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.110211 23208 tablet_service.cc:1467] Processing CreateTablet for tablet bbc3e9de57f1419aa78b468e7d684a89 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260811 02:59:23.111152 23062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc3e9de57f1419aa78b468e7d684a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.111892 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc3e9de57f1419aa78b468e7d684a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.112315 23349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c377df55c8049f6bd13fe8f53f52b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.113145 23350 tablet_service.cc:1467] Processing CreateTablet for tablet fb6662cff60c4865acd527a45a257580 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260811 02:59:23.113754 23350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6662cff60c4865acd527a45a257580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.116559 23207 tablet_service.cc:1467] Processing CreateTablet for tablet fb6662cff60c4865acd527a45a257580 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260811 02:59:23.117167 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6662cff60c4865acd527a45a257580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.118898 23206 tablet_service.cc:1467] Processing CreateTablet for tablet 7c377df55c8049f6bd13fe8f53f52b28 (DEFAULT_TABLE table=all-types-table [id=7e267dfa01d84cccb877a65368c4791a]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260811 02:59:23.119436 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c377df55c8049f6bd13fe8f53f52b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.122728 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c377df55c8049f6bd13fe8f53f52b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:59:23.134122 23444 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: Bootstrap starting.
I20260811 02:59:23.138880 23444 tablet_bootstrap.cc:654] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.140292 23444 log.cc:824] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:23.147277 23445 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: Bootstrap starting.
I20260811 02:59:23.149725 23443 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: Bootstrap starting.
I20260811 02:59:23.151868 23444 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: No bootstrap required, opened a new log
I20260811 02:59:23.152190 23444 ts_tablet_manager.cc:1397] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20260811 02:59:23.153245 23445 tablet_bootstrap.cc:654] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.153586 23443 tablet_bootstrap.cc:654] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.154748 23445 log.cc:824] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:23.154896 23443 log.cc:824] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: Log is configured to *not* fsync() on all Append() calls
I20260811 02:59:23.158197 23445 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: No bootstrap required, opened a new log
I20260811 02:59:23.158546 23445 ts_tablet_manager.cc:1397] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260811 02:59:23.158591 23443 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: No bootstrap required, opened a new log
I20260811 02:59:23.158886 23443 ts_tablet_manager.cc:1397] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260811 02:59:23.162701 23444 raft_consensus.cc:348] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.163228 23444 raft_consensus.cc:374] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.163335 23444 raft_consensus.cc:729] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c17323c7214a8d8dbb4f2e000a05c8, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.164062 23444 consensus_queue.cc:260] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.166121 23444 ts_tablet_manager.cc:1428] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260811 02:59:23.166532 23444 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8: Bootstrap starting.
I20260811 02:59:23.168426 23444 tablet_bootstrap.cc:654] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.168684 23445 raft_consensus.cc:348] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.169188 23445 raft_consensus.cc:374] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.169303 23445 raft_consensus.cc:729] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5264f57db51340f9900ee8ceb7998983, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.169672 23443 raft_consensus.cc:348] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.170019 23445 consensus_queue.cc:260] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.170243 23443 raft_consensus.cc:374] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.170373 23443 raft_consensus.cc:729] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbab65af5824618b363b5a8b389be33, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.171238 23443 consensus_queue.cc:260] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.174536 23281 heartbeater.cc:503] Master 127.21.184.254:33799 was elected leader, sending a full tablet report...
I20260811 02:59:23.175351 23445 ts_tablet_manager.cc:1428] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260811 02:59:23.175752 23445 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: Bootstrap starting.
I20260811 02:59:23.176738 23443 ts_tablet_manager.cc:1428] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260811 02:59:23.177179 23424 heartbeater.cc:503] Master 127.21.184.254:33799 was elected leader, sending a full tablet report...
I20260811 02:59:23.177744 23445 tablet_bootstrap.cc:654] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.184391 23443 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: Bootstrap starting.
I20260811 02:59:23.185329 23449 raft_consensus.cc:482] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:23.185704 23449 raft_consensus.cc:504] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.186466 23443 tablet_bootstrap.cc:654] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.188171 23449 leader_election.cc:288] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825), 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255)
I20260811 02:59:23.189138 23445 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: No bootstrap required, opened a new log
I20260811 02:59:23.189343 23445 ts_tablet_manager.cc:1397] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260811 02:59:23.191038 23443 tablet_bootstrap.cc:492] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: No bootstrap required, opened a new log
I20260811 02:59:23.190822 23445 raft_consensus.cc:348] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.191155 23445 raft_consensus.cc:374] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.191287 23445 raft_consensus.cc:729] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5264f57db51340f9900ee8ceb7998983, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.191596 23443 ts_tablet_manager.cc:1397] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 02:59:23.191622 23445 consensus_queue.cc:260] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.192286 23445 ts_tablet_manager.cc:1428] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260811 02:59:23.193166 23443 raft_consensus.cc:348] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.193588 23443 raft_consensus.cc:374] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.193789 23443 raft_consensus.cc:729] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbab65af5824618b363b5a8b389be33, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.194173 23443 consensus_queue.cc:260] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.194798 23443 ts_tablet_manager.cc:1428] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260811 02:59:23.194947 23444 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8: No bootstrap required, opened a new log
I20260811 02:59:23.195123 23444 ts_tablet_manager.cc:1397] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.021s
I20260811 02:59:23.196216 23444 raft_consensus.cc:348] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.196528 23444 raft_consensus.cc:374] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.196666 23444 raft_consensus.cc:729] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c17323c7214a8d8dbb4f2e000a05c8, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.196794 23445 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: Bootstrap starting.
I20260811 02:59:23.196919 23444 consensus_queue.cc:260] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.199400 23445 tablet_bootstrap.cc:654] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.201712 23443 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: Bootstrap starting.
I20260811 02:59:23.203233 23444 ts_tablet_manager.cc:1428] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260811 02:59:23.203879 23443 tablet_bootstrap.cc:654] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.206576 23444 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: Bootstrap starting.
I20260811 02:59:23.209179 23444 tablet_bootstrap.cc:654] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: Neither blocks nor log segments found. Creating new log.
I20260811 02:59:23.212932 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6662cff60c4865acd527a45a257580" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33" is_pre_election: true
I20260811 02:59:23.214416 23375 raft_consensus.cc:2454] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 0.
I20260811 02:59:23.214749 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6662cff60c4865acd527a45a257580" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5264f57db51340f9900ee8ceb7998983" is_pre_election: true
I20260811 02:59:23.215342 23233 raft_consensus.cc:2454] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 0.
I20260811 02:59:23.215308 23012 leader_election.cc:302] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbab65af5824618b363b5a8b389be33, f4c17323c7214a8d8dbb4f2e000a05c8; no voters:
I20260811 02:59:23.215843 23449 raft_consensus.cc:2781] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:23.216008 23449 raft_consensus.cc:482] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:23.216140 23449 raft_consensus.cc:3037] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.216655 23445 tablet_bootstrap.cc:492] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: No bootstrap required, opened a new log
I20260811 02:59:23.216840 23445 ts_tablet_manager.cc:1397] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.014s
I20260811 02:59:23.217682 23445 raft_consensus.cc:348] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.217939 23445 raft_consensus.cc:374] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.218029 23445 raft_consensus.cc:729] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5264f57db51340f9900ee8ceb7998983, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.218380 23445 consensus_queue.cc:260] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.219367 23449 raft_consensus.cc:504] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.221096 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6662cff60c4865acd527a45a257580" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5264f57db51340f9900ee8ceb7998983"
I20260811 02:59:23.221323 23233 raft_consensus.cc:3037] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.221668 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6662cff60c4865acd527a45a257580" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33"
I20260811 02:59:23.221920 23375 raft_consensus.cc:3037] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.224601 23233 raft_consensus.cc:2454] T fb6662cff60c4865acd527a45a257580 P 5264f57db51340f9900ee8ceb7998983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 1.
I20260811 02:59:23.225159 23375 raft_consensus.cc:2454] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 1.
I20260811 02:59:23.225674 23013 leader_election.cc:302] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4cbab65af5824618b363b5a8b389be33, 5264f57db51340f9900ee8ceb7998983, f4c17323c7214a8d8dbb4f2e000a05c8; no voters:
I20260811 02:59:23.225802 23449 leader_election.cc:288] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 election: Requested vote from peers 5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825), 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255)
I20260811 02:59:23.226145 23449 raft_consensus.cc:2781] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:23.226321 23449 raft_consensus.cc:686] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 LEADER]: Becoming Leader. State: Replica: f4c17323c7214a8d8dbb4f2e000a05c8, State: Running, Role: LEADER
I20260811 02:59:23.226990 23449 consensus_queue.cc:237] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.232246 23445 ts_tablet_manager.cc:1428] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20260811 02:59:23.235224 22931 catalog_manager.cc:5495] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193). New cstate: current_term: 1 leader_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } health_report { overall_health: UNKNOWN } } }
I20260811 02:59:23.242434 23443 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: No bootstrap required, opened a new log
I20260811 02:59:23.242659 23443 ts_tablet_manager.cc:1397] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.034s
I20260811 02:59:23.243104 23444 tablet_bootstrap.cc:492] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: No bootstrap required, opened a new log
I20260811 02:59:23.243309 23444 ts_tablet_manager.cc:1397] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.024s
I20260811 02:59:23.243449 23443 raft_consensus.cc:348] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.243690 23443 raft_consensus.cc:374] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.243779 23443 raft_consensus.cc:729] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbab65af5824618b363b5a8b389be33, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.244061 23444 raft_consensus.cc:348] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.244115 23443 consensus_queue.cc:260] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.244289 23444 raft_consensus.cc:374] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:59:23.244374 23444 raft_consensus.cc:729] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c17323c7214a8d8dbb4f2e000a05c8, State: Initialized, Role: FOLLOWER
I20260811 02:59:23.244655 23443 ts_tablet_manager.cc:1428] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 02:59:23.244645 23444 consensus_queue.cc:260] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.245222 23444 ts_tablet_manager.cc:1428] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260811 02:59:23.264655 23138 tablet.cc:2369] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:59:23.271536 23282 tablet.cc:2369] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:59:23.278955 23425 tablet.cc:2369] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:59:23.360001 23451 raft_consensus.cc:482] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:23.360348 23451 raft_consensus.cc:504] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.362203 23451 leader_election.cc:288] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255), f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161)
I20260811 02:59:23.372248 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc3e9de57f1419aa78b468e7d684a89" candidate_uuid: "5264f57db51340f9900ee8ceb7998983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33" is_pre_election: true
I20260811 02:59:23.372578 23375 raft_consensus.cc:2454] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5264f57db51340f9900ee8ceb7998983 in term 0.
I20260811 02:59:23.372898 23084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc3e9de57f1419aa78b468e7d684a89" candidate_uuid: "5264f57db51340f9900ee8ceb7998983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" is_pre_election: true
I20260811 02:59:23.373096 23158 leader_election.cc:302] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbab65af5824618b363b5a8b389be33, 5264f57db51340f9900ee8ceb7998983; no voters:
I20260811 02:59:23.373364 23084 raft_consensus.cc:2454] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5264f57db51340f9900ee8ceb7998983 in term 0.
I20260811 02:59:23.373564 23451 raft_consensus.cc:2781] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:23.373749 23451 raft_consensus.cc:482] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:23.373872 23451 raft_consensus.cc:3037] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.375891 23451 raft_consensus.cc:504] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.376540 23451 leader_election.cc:288] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [CANDIDATE]: Term 1 election: Requested vote from peers 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255), f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161)
I20260811 02:59:23.376992 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc3e9de57f1419aa78b468e7d684a89" candidate_uuid: "5264f57db51340f9900ee8ceb7998983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33"
I20260811 02:59:23.377142 23084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc3e9de57f1419aa78b468e7d684a89" candidate_uuid: "5264f57db51340f9900ee8ceb7998983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8"
I20260811 02:59:23.377242 23375 raft_consensus.cc:3037] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.377372 23084 raft_consensus.cc:3037] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.380321 23375 raft_consensus.cc:2454] T bbc3e9de57f1419aa78b468e7d684a89 P 4cbab65af5824618b363b5a8b389be33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5264f57db51340f9900ee8ceb7998983 in term 1.
I20260811 02:59:23.380376 23084 raft_consensus.cc:2454] T bbc3e9de57f1419aa78b468e7d684a89 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5264f57db51340f9900ee8ceb7998983 in term 1.
I20260811 02:59:23.380880 23460 raft_consensus.cc:482] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:59:23.380854 23157 leader_election.cc:302] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5264f57db51340f9900ee8ceb7998983, f4c17323c7214a8d8dbb4f2e000a05c8; no voters:
I20260811 02:59:23.381058 23460 raft_consensus.cc:504] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.381352 23451 raft_consensus.cc:2781] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:23.381898 23451 raft_consensus.cc:686] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [term 1 LEADER]: Becoming Leader. State: Replica: 5264f57db51340f9900ee8ceb7998983, State: Running, Role: LEADER
I20260811 02:59:23.382380 23460 leader_election.cc:288] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825), 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255)
I20260811 02:59:23.382503 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c377df55c8049f6bd13fe8f53f52b28" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5264f57db51340f9900ee8ceb7998983" is_pre_election: true
I20260811 02:59:23.382596 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c377df55c8049f6bd13fe8f53f52b28" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33" is_pre_election: true
I20260811 02:59:23.382818 23233 raft_consensus.cc:2454] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 0.
I20260811 02:59:23.382917 23375 raft_consensus.cc:2454] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 0.
I20260811 02:59:23.383327 23013 leader_election.cc:302] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4cbab65af5824618b363b5a8b389be33, 5264f57db51340f9900ee8ceb7998983, f4c17323c7214a8d8dbb4f2e000a05c8; no voters:
I20260811 02:59:23.383628 23460 raft_consensus.cc:2781] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:59:23.383744 23460 raft_consensus.cc:482] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:59:23.383823 23460 raft_consensus.cc:3037] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.384027 23451 consensus_queue.cc:237] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } }
I20260811 02:59:23.386667 23460 raft_consensus.cc:504] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.387298 23460 leader_election.cc:288] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 election: Requested vote from peers 5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825), 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255)
I20260811 02:59:23.387895 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c377df55c8049f6bd13fe8f53f52b28" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbab65af5824618b363b5a8b389be33"
I20260811 02:59:23.387903 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c377df55c8049f6bd13fe8f53f52b28" candidate_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5264f57db51340f9900ee8ceb7998983"
I20260811 02:59:23.388084 23375 raft_consensus.cc:3037] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.388134 23233 raft_consensus.cc:3037] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:59:23.389313 22930 catalog_manager.cc:5495] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5264f57db51340f9900ee8ceb7998983 (127.21.184.194). New cstate: current_term: 1 leader_uuid: "5264f57db51340f9900ee8ceb7998983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } health_report { overall_health: UNKNOWN } } }
I20260811 02:59:23.390357 23375 raft_consensus.cc:2454] T 7c377df55c8049f6bd13fe8f53f52b28 P 4cbab65af5824618b363b5a8b389be33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 1.
I20260811 02:59:23.390780 23012 leader_election.cc:302] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbab65af5824618b363b5a8b389be33, f4c17323c7214a8d8dbb4f2e000a05c8; no voters:
I20260811 02:59:23.390885 23233 raft_consensus.cc:2454] T 7c377df55c8049f6bd13fe8f53f52b28 P 5264f57db51340f9900ee8ceb7998983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c17323c7214a8d8dbb4f2e000a05c8 in term 1.
I20260811 02:59:23.391057 23460 raft_consensus.cc:2781] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:59:23.391206 23460 raft_consensus.cc:686] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [term 1 LEADER]: Becoming Leader. State: Replica: f4c17323c7214a8d8dbb4f2e000a05c8, State: Running, Role: LEADER
I20260811 02:59:23.391506 23460 consensus_queue.cc:237] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } }
I20260811 02:59:23.395918 22930 catalog_manager.cc:5495] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193). New cstate: current_term: 1 leader_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 } health_report { overall_health: UNKNOWN } } }
I20260811 02:59:23.610291 23460 consensus_queue.cc:1035] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:23.620584 23449 consensus_queue.cc:1035] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:23.823904 23449 consensus_queue.cc:1035] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:23.829927 23449 consensus_queue.cc:1035] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5264f57db51340f9900ee8ceb7998983" member_type: VOTER last_known_addr { host: "127.21.184.194" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:23.927306 23451 consensus_queue.cc:1035] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cbab65af5824618b363b5a8b389be33" member_type: VOTER last_known_addr { host: "127.21.184.195" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:23.937703 23451 consensus_queue.cc:1035] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c17323c7214a8d8dbb4f2e000a05c8" member_type: VOTER last_known_addr { host: "127.21.184.193" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:59:25.019485 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling FlushMRSOp(fb6662cff60c4865acd527a45a257580): perf score=1.000000
W20260811 02:59:25.042670 23428 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.21.184.193:43161, user_credentials={real_user=slave}} blocked reactor thread for 69364.5us
W20260811 02:59:25.075943 23497 log.cc:924] Time spent T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983: Append to log took a long time: real 0.062s user 0.004s sys 0.000s
I20260811 02:59:25.535275 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling FlushMRSOp(bbc3e9de57f1419aa78b468e7d684a89): perf score=1.937278
W20260811 02:59:25.736047 23430 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.21.184.194:45825, user_credentials={real_user=slave}} blocked reactor thread for 151093us
I20260811 02:59:25.814183 23425 maintenance_manager.cc:419] P 4cbab65af5824618b363b5a8b389be33: Scheduling FlushMRSOp(fb6662cff60c4865acd527a45a257580): perf score=1.000000
W20260811 02:59:26.025005 23306 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-2/data/data (489398272 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 02:59:26.143100 23306 tablet_replica_mm_ops.cc:214] T fb6662cff60c4865acd527a45a257580 P 4cbab65af5824618b363b5a8b389be33: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet fb6662cff60c4865acd527a45a257580: unable to open output file for column double_val DOUBLE NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in fb6662cff60c4865acd527a45a257580's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 02:59:26.143606 23306 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet fb6662cff60c4865acd527a45a257580: unable to open output file for column double_val DOUBLE NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in fb6662cff60c4865acd527a45a257580's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fb672c8d9ad google::LogMessage::Fail() at ??:0
@ 0x7fb672c9088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb672c8d4de google::LogMessage::Flush() at ??:0
@ 0x7fb672c8ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb682062e7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fb67577b090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fb67577de53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fb67577dce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb6758d9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6758dc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb6758dc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb6758b7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb678b916db start_thread at ??:0
@ 0x7fb671b2761f clone at ??:0
W20260811 02:59:26.821409 23158 connection.cc:535] client connection to 127.21.184.195:35255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:59:26.821447 23012 connection.cc:535] client connection to 127.21.184.195:35255 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:59:26.821785 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:59:26.821775 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:59:26.861802 23552 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:59:26.827464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.184.195:35255 (local address 127.21.184.194:48869)
0811 02:59:26.856612 (+ 29148us) negotiation.cc:107] Waiting for socket to connect
0811 02:59:26.856890 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":28945,"thread_start_us":29568,"threads_started":1}
W20260811 02:59:26.862615 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20260811 02:59:26.875464 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260811 02:59:26.875865 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 02:59:27.370764 23524 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48462 (ReqId={client: 802ea6ba498640768ecf41ec3c07ebb5, seq_no=3, attempt_no=0}) took 1135 ms. Trace:
I20260811 02:59:27.371132 23524 rpcz_store.cc:276] 0811 02:59:26.235306 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 02:59:26.235421 (+ 115us) service_pool.cc:227] Handling call
0811 02:59:27.370694 (+1135273us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0811 02:59:26.237297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet fb6662cff60c4865acd527a45a257580
0811 02:59:26.237521 (+ 224us) write_op.cc:432] Acquiring schema lock in shared mode
0811 02:59:26.237528 (+ 7us) write_op.cc:435] Acquired schema lock
0811 02:59:26.237531 (+ 3us) tablet.cc:660] Decoding operations
0811 02:59:26.582180 (+344649us) write_op.cc:620] Acquiring the partition lock for write op
0811 02:59:26.582198 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0811 02:59:26.582205 (+ 7us) tablet.cc:683] Acquiring locks for 4886 operations
0811 02:59:26.742165 (+159960us) tablet.cc:699] Row locks acquired
0811 02:59:26.742170 (+ 5us) write_op.cc:260] PREPARE: finished
0811 02:59:26.742263 (+ 93us) write_op.cc:270] Start()
0811 02:59:26.742319 (+ 56us) write_op.cc:276] Timestamp: P: 1786417166742220 usec, L: 0
0811 02:59:26.742324 (+ 5us) op_driver.cc:331] REPLICATION: starting
0811 02:59:26.743683 (+ 1359us) log.cc:842] Serialized 772632 byte log entry
0811 02:59:26.813380 (+ 69697us) op_driver.cc:444] REPLICATION: finished
0811 02:59:26.813517 (+ 137us) write_op.cc:301] APPLY: starting
0811 02:59:26.813541 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0811 02:59:26.871902 (+ 58361us) tablet.cc:1368] finished BulkCheckPresence
0811 02:59:26.871948 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0811 02:59:27.339815 (+467867us) tablet.cc:1381] finished ApplyRowOperation cycle
0811 02:59:27.342673 (+ 2858us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0811 02:59:27.342682 (+ 9us) write_op.cc:312] APPLY: finished
0811 02:59:27.362858 (+ 20176us) log.cc:842] Serialized 29338 byte log entry
0811 02:59:27.362994 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0811 02:59:27.370211 (+ 7217us) write_op.cc:454] Released schema lock
0811 02:59:27.370514 (+ 303us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":4886,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":127109,"prepare.run_wall_time_us":538813,"raft.queue_time_us":65,"raft.run_cpu_time_us":1547,"raft.run_wall_time_us":1547,"replication_time_us":71019,"thread_start_us":22210,"threads_started":1,"wal-append.queue_time_us":10240}]]}
W20260811 02:59:27.452884 23428 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.21.184.193:43161, user_credentials={real_user=slave}} blocked reactor thread for 79051.2us
I20260811 02:59:27.583007 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: FlushMRSOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 2.562s user 0.774s sys 0.124s Metrics: {"bytes_written":237318,"cfile_init":1,"compiler_manager_pool.queue_time_us":6768,"dirs.queue_time_us":7077,"dirs.run_cpu_time_us":338,"dirs.run_wall_time_us":4797,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4630,"lbm_writes_lt_1ms":81,"peak_mem_usage":0,"rows_written":5114,"thread_start_us":18500,"threads_started":2}
I20260811 02:59:27.584862 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling FlushMRSOp(bbc3e9de57f1419aa78b468e7d684a89): perf score=1.937278
I20260811 02:59:28.088526 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: FlushMRSOp(bbc3e9de57f1419aa78b468e7d684a89) complete. Timing: real 2.552s user 1.350s sys 0.164s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":6743,"compiler_manager_pool.run_cpu_time_us":1048965,"compiler_manager_pool.run_wall_time_us":1989452,"dirs.queue_time_us":17292,"dirs.run_cpu_time_us":318,"dirs.run_wall_time_us":1009,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":18460,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":85,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":238336,"thread_start_us":40281,"threads_started":3,"wal-append.queue_time_us":12883}
I20260811 02:59:28.090374 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling FlushMRSOp(fb6662cff60c4865acd527a45a257580): perf score=1.937278
W20260811 02:59:28.340528 23428 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.21.184.193:43161, user_credentials={real_user=slave}} blocked reactor thread for 155047us
I20260811 02:59:28.475473 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:28.488881 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:28.542816 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:28.841430 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:28.843619 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:28.893913 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:28.939462 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: FlushMRSOp(bbc3e9de57f1419aa78b468e7d684a89) complete. Timing: real 1.354s user 1.149s sys 0.094s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":289883,"compiler_manager_pool.run_cpu_time_us":65,"compiler_manager_pool.run_wall_time_us":65,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":2021,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4165,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":631,"threads_started":2,"wal-append.queue_time_us":460}
I20260811 02:59:28.940495 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling FlushMRSOp(7c377df55c8049f6bd13fe8f53f52b28): perf score=1.937278
W20260811 02:59:29.017482 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260811 02:59:29.128026 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 02:59:29.179266 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: FlushMRSOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 1.089s user 0.914s sys 0.117s Metrics: {"bytes_written":472622,"cfile_init":1,"dirs.queue_time_us":481,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4715,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":583,"threads_started":2,"wal-append.queue_time_us":392}
I20260811 02:59:29.180203 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling FlushMRSOp(7c377df55c8049f6bd13fe8f53f52b28): perf score=1.937278
I20260811 02:59:29.219872 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:29.230993 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:29.274325 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:29.319124 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 02:59:29.752183 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:29.770856 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:29.812856 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:29.849171 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: FlushMRSOp(7c377df55c8049f6bd13fe8f53f52b28) complete. Timing: real 0.908s user 0.799s sys 0.099s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":390,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1598,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3857,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":534,"threads_started":2,"wal-append.queue_time_us":405}
I20260811 02:59:29.850260 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling UndoDeltaBlockGCOp(7c377df55c8049f6bd13fe8f53f52b28): 63437 bytes on disk
I20260811 02:59:29.851076 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: UndoDeltaBlockGCOp(7c377df55c8049f6bd13fe8f53f52b28) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20260811 02:59:29.851892 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling FlushMRSOp(fb6662cff60c4865acd527a45a257580): perf score=1.000000
I20260811 02:59:30.041271 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: FlushMRSOp(7c377df55c8049f6bd13fe8f53f52b28) complete. Timing: real 0.861s user 0.795s sys 0.040s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":458,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1394,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3664,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":5376,"thread_start_us":612,"threads_started":2,"wal-append.queue_time_us":408}
I20260811 02:59:30.042131 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling UndoDeltaBlockGCOp(7c377df55c8049f6bd13fe8f53f52b28): 63437 bytes on disk
I20260811 02:59:30.042949 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: UndoDeltaBlockGCOp(7c377df55c8049f6bd13fe8f53f52b28) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260811 02:59:30.043576 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580): 63422 bytes on disk
I20260811 02:59:30.044107 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260811 02:59:30.044617 23282 maintenance_manager.cc:419] P 5264f57db51340f9900ee8ceb7998983: Scheduling UndoDeltaBlockGCOp(bbc3e9de57f1419aa78b468e7d684a89): 62923 bytes on disk
I20260811 02:59:30.045077 23162 maintenance_manager.cc:643] P 5264f57db51340f9900ee8ceb7998983: UndoDeltaBlockGCOp(bbc3e9de57f1419aa78b468e7d684a89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20260811 02:59:30.169070 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: FlushMRSOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 0.317s user 0.277s sys 0.036s Metrics: {"bytes_written":220984,"cfile_init":1,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1513,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2319,"lbm_writes_lt_1ms":81,"peak_mem_usage":0,"rows_written":4886,"thread_start_us":261,"threads_started":1,"wal-append.queue_time_us":424}
I20260811 02:59:30.170540 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580): 63080 bytes on disk
I20260811 02:59:30.171427 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":137,"lbm_reads_lt_1ms":8}
I20260811 02:59:30.172183 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling CompactRowSetsOp(fb6662cff60c4865acd527a45a257580): perf score=0.014860
I20260811 02:59:30.447999 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:30.455443 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:30.494771 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:30.668320 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: CompactRowSetsOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 0.496s user 0.487s sys 0.008s Metrics: {"bytes_written":472622,"cfile_cache_hit":40,"cfile_cache_hit_bytes":398375,"cfile_cache_miss":116,"cfile_cache_miss_bytes":590507,"cfile_init":31,"delta_iterators_relevant":4,"dirs.queue_time_us":3333,"dirs.run_cpu_time_us":1785,"dirs.run_wall_time_us":8937,"drs_written":1,"lbm_read_time_us":3628,"lbm_reads_lt_1ms":240,"lbm_write_time_us":3333,"lbm_writes_lt_1ms":86,"mutex_wait_us":152,"num_input_rowsets":2,"peak_mem_usage":108216,"rows_written":10000,"spinlock_wait_cycles":13696,"thread_start_us":2076,"threads_started":8}
I20260811 02:59:30.669541 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580): 63422 bytes on disk
I20260811 02:59:30.670262 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: UndoDeltaBlockGCOp(fb6662cff60c4865acd527a45a257580) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20260811 02:59:30.670943 23138 maintenance_manager.cc:419] P f4c17323c7214a8d8dbb4f2e000a05c8: Scheduling UndoDeltaBlockGCOp(bbc3e9de57f1419aa78b468e7d684a89): 62923 bytes on disk
I20260811 02:59:30.671458 23015 maintenance_manager.cc:643] P f4c17323c7214a8d8dbb4f2e000a05c8: UndoDeltaBlockGCOp(bbc3e9de57f1419aa78b468e7d684a89) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20260811 02:59:31.345852 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:31.349253 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260811 02:59:31.374627 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:31.390951 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:31.572176 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260811 02:59:31.689524 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260811 02:59:32.111359 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 02:59:32.200743 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20260811 02:59:32.489056 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:32.489151 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:32.521581 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:33.611500 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:33.611495 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:33.645955 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:33.702484 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260811 02:59:33.875701 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260811 02:59:34.244019 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260811 02:59:34.734222 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:34.734556 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:34.770617 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:35.859251 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:35.859910 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:35.893676 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:36.364298 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260811 02:59:36.578002 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260811 02:59:36.757463 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 02:59:36.996443 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:36.996433 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:37.034615 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:37.193714 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 02:59:37.311365 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 02:59:38.120375 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:38.121934 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:38.157689 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:38.734256 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260811 02:59:39.203392 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 02:59:39.253784 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:39.255564 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:39.292073 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:39.410748 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 02:59:40.381227 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:40.382517 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:40.417685 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:41.252516 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260811 02:59:41.393837 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 02:59:41.511765 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:41.522059 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:41.559341 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:41.826604 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260811 02:59:42.360157 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 02:59:42.605062 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260811 02:59:42.654706 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:42.654995 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:42.693110 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:43.688467 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 02:59:43.790714 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:43.796784 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:43.830286 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:43.986636 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260811 02:59:44.410573 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 02:59:44.930506 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:44.931032 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:44.977128 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 02:59:46.068027 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:46.068930 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:46.101548 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:46.375244 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260811 02:59:46.440856 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260811 02:59:46.991338 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260811 02:59:47.203060 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:47.203281 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:47.236306 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:47.532737 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 02:59:47.977882 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 02:59:48.336630 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:48.336776 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:48.373041 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:48.897612 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260811 02:59:49.061926 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 02:59:49.479082 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:49.479174 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:49.514586 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:49.617511 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 02:59:50.610786 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:50.617079 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:50.651742 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:51.213006 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 02:59:51.465081 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260811 02:59:51.742237 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:51.743108 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:51.793718 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:52.053172 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 02:59:52.539252 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260811 02:59:52.885522 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:52.885571 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:52.916992 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:53.131444 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 02:59:53.584800 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 02:59:54.008947 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:54.008982 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:54.050064 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:54.051229 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260811 02:59:54.225584 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 02:59:55.134678 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:55.136932 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:55.171147 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:56.094419 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 02:59:56.260421 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:56.260924 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:56.293756 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:56.567546 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260811 02:59:56.568799 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 02:59:57.397027 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:57.400012 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:57.439368 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:57.568323 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 02:59:58.184285 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 02:59:58.541495 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 02:59:58.542474 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:58.580487 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 02:59:58.876617 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260811 02:59:59.193488 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260811 02:59:59.376343 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260811 02:59:59.680823 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 02:59:59.681069 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 02:59:59.719607 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 03:00:00.815879 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:00.815886 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:00.858358 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:01.164311 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260811 03:00:01.743096 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260811 03:00:01.932080 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 03:00:01.952750 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:01.953045 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:02.001138 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:02.659811 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260811 03:00:03.102533 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:03.105458 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:03.142271 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:03.522696 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:00:03.554316 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260811 03:00:04.027204 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 03:00:04.245396 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:04.252279 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:04.289429 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:04.606258 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 03:00:05.406595 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:05.407176 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:05.440062 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:05.867650 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260811 03:00:06.548012 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:06.556489 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:06.599413 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:06.660117 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 03:00:07.494084 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 03:00:07.668642 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260811 03:00:07.698179 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:07.700549 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:07.744884 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:08.060650 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 03:00:08.837754 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:08.837812 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:08.876318 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:09.043936 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:00:09.061024 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 03:00:09.984438 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:09.985525 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:10.027294 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:10.041311 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260811 03:00:10.641673 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 03:00:11.133914 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:11.134369 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:11.182050 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:11.539602 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 03:00:12.273319 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:12.276787 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:12.320623 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:12.604387 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260811 03:00:13.017680 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:00:13.077901 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260811 03:00:13.419036 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:13.419067 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:13.455444 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:13.961577 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 03:00:14.230829 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 03:00:14.547129 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:14.548301 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:14.592724 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:15.174080 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 03:00:15.582553 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 03:00:15.681039 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:15.683115 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:15.726538 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:16.354339 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 03:00:16.821733 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:16.821877 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:16.857990 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:17.881560 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 03:00:17.948154 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:17.950250 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:17.987532 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:18.123332 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260811 03:00:18.313860 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:00:18.981580 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 03:00:19.079841 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:19.079869 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:19.125877 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:19.613392 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 03:00:20.219998 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:20.225520 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:20.258841 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:20.385841 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260811 03:00:20.486121 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 03:00:21.348737 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:21.348737 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:21.397250 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:21.695175 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260811 03:00:22.500020 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:22.504804 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:22.548331 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:22.778669 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 03:00:23.215503 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260811 03:00:23.390311 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260811 03:00:23.648844 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:23.649363 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:23.684253 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:24.484447 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 03:00:24.737509 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 03:00:24.777691 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:24.777936 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:24.813426 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:25.164887 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260811 03:00:25.824448 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260811 03:00:25.919119 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:25.920426 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:25.975349 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:26.664546 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 03:00:27.075727 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:27.078586 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:27.110777 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:27.825258 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260811 03:00:28.184409 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 03:00:28.201996 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:28.202744 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:28.238616 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:28.769603 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260811 03:00:29.332674 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:29.332690 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:29.368949 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:29.409463 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260811 03:00:29.781107 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:00:30.182854 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260811 03:00:30.461504 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:30.461658 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:30.502974 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:30.805877 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260811 03:00:31.593073 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:31.598945 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:31.631727 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:31.737104 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260811 03:00:32.718602 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 03:00:32.726212 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:32.728287 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:32.773111 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:33.121443 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260811 03:00:33.872946 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:33.876788 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:33.909710 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:34.040167 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:00:34.041188 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 03:00:35.005319 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:35.005319 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260811 03:00:35.033578 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:35.048098 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:35.131889 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260811 03:00:35.634644 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 03:00:36.140853 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:36.141512 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:36.175717 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:36.477411 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260811 03:00:37.263615 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:37.264514 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:37.295795 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:37.710460 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 03:00:38.163938 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260811 03:00:38.385133 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:38.389086 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:38.437803 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:38.783420 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 03:00:39.266947 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20260811 03:00:39.530952 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:39.531255 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:39.577332 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:40.284972 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260811 03:00:40.291985 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260811 03:00:40.677747 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:40.684348 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:40.718142 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:40.834079 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 03:00:41.363571 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260811 03:00:41.821727 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:41.821712 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:41.857167 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:42.449196 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 03:00:42.948599 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:42.949605 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:42.996224 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:43.641340 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260811 03:00:43.853905 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260811 03:00:44.087872 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:44.097891 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:44.130573 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:44.290815 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:00:44.807370 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 03:00:45.224436 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:45.225329 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:45.259337 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:45.875671 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:00:46.100869 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260811 03:00:46.347496 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 03:00:46.353832 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:46.356683 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:46.400719 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:47.095515 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 03:00:47.497816 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:47.502131 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:47.543627 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:48.434540 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 03:00:48.638813 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:48.639012 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:48.689640 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:48.865923 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260811 03:00:49.493440 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:00:49.494714 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260811 03:00:49.779707 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:49.781469 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:49.814380 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 03:00:50.922276 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:50.922399 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:50.956632 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:51.209455 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 03:00:51.355949 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:00:51.357482 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260811 03:00:51.867244 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 03:00:52.054457 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:52.061110 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:52.094827 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 03:00:53.188969 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:53.195041 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:53.235718 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:53.596653 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 03:00:53.688467 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 03:00:54.329814 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:54.329881 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:54.364683 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:54.530164 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:00:54.531275 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 03:00:55.467829 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:55.475327 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:55.511947 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:55.987872 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260811 03:00:55.991082 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260811 03:00:56.462229 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260811 03:00:56.609018 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:00:56.609800 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:56.648631 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:57.160857 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260811 03:00:57.766048 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:57.769038 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:57.812235 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:58.346930 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260811 03:00:58.499811 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 03:00:58.903162 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:00:58.903398 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:00:58.947280 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:00:59.754671 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:00:59.755662 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 03:01:00.046532 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:00.056914 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:00.093008 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:00.861986 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260811 03:01:00.867887 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 03:01:01.190304 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:01:01.190615 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:01.228111 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:01.822347 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20260811 03:01:02.332823 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:02.333230 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:02.377323 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:02.612047 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260811 03:01:03.347630 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260811 03:01:03.374411 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 03:01:03.469161 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:03.474462 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:03.508728 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 03:01:04.620106 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:04.620267 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:04.666685 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:04.761685 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:05.057960 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260811 03:01:05.759027 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:01:05.759310 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260811 03:01:05.776319 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:05.794234 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:05.947293 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 03:01:06.901082 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:06.901340 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:06.935243 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:07.105842 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:07.534101 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 03:01:08.020674 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:08.023465 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:08.071364 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:08.351604 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260811 03:01:08.669822 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260811 03:01:09.164903 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:01:09.165925 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:09.205688 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:09.792853 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:09.910725 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260811 03:01:10.324060 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:10.329521 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:10.367781 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:10.670912 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 03:01:11.245846 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 03:01:11.489418 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:11.493348 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:11.531975 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:12.205703 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:12.512444 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 03:01:12.634430 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:12.635499 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:12.681270 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:12.960037 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 03:01:13.783185 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:13.789358 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:13.823519 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:13.829401 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 03:01:14.583422 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260811 03:01:14.875409 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20260811 03:01:14.917377 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:01:14.918007 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:14.951943 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:15.368240 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 03:01:16.055540 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:16.059231 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:16.094362 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:16.222157 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 03:01:17.184669 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260811 03:01:17.185089 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.193:43161, tserver@127.21.184.194:45825
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
W20260811 03:01:17.229318 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260811 03:01:17.231911 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:17.623770 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:01:18.145722 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260811 03:01:18.319172 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:18.319923 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:18.362588 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:18.714809 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 03:01:19.468747 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:19.472462 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:19.506503 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:19.556999 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 03:01:20.039702 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:20.416384 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260811 03:01:20.617509 23207 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 03:01:20.617825 23061 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
37817f3438924819a8b313e319f2be3c | 127.21.184.254:33799 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.21.184.212:34139 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+-------------+----------+----------------+-----------------
5264f57db51340f9900ee8ceb7998983 | 127.21.184.194:45825 | HEALTHY | <none> | 1 | 0
f4c17323c7214a8d8dbb4f2e000a05c8 | 127.21.184.193:43161 | HEALTHY | <none> | 2 | 0
4cbab65af5824618b363b5a8b389be33 | 127.21.184.195:35255 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.21.184.195:35255: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.21.184.194:45825, 127.21.184.193:43161
ipki_server_key_size | 768 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.193 | experimental | 127.21.184.193:43161
local_ip_for_outbound_sockets | 127.21.184.194 | experimental | 127.21.184.194:45825
never_fsync | true | unsafe,advanced | 127.21.184.194:45825, 127.21.184.193:43161
openssl_security_level_override | 0 | unsafe,hidden | 127.21.184.194:45825, 127.21.184.193:43161
rpc_server_allow_ephemeral_ports | true | unsafe | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_format | pb | hidden | 127.21.184.194:45825, 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.184.193:43161
server_dump_info_path | /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.184.194:45825
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.21.184.194:45825, 127.21.184.193:43161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+--------------------------------------------
builtin_ntp_servers | 127.21.184.212:34139 | 127.21.184.194:45825, 127.21.184.193:43161
time_source | builtin | 127.21.184.194:45825, 127.21.184.193:43161
Version Summary
Version | Servers
-----------------+-----------------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.21.184.254:33799, tserver@127.21.184.194:45825, tserver@127.21.184.193:43161
Tablet Summary
Tablet bbc3e9de57f1419aa78b468e7d684a89 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING [LEADER]
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING
All reported replicas are:
A = 5264f57db51340f9900ee8ceb7998983
B = 4cbab65af5824618b363b5a8b389be33
C = f4c17323c7214a8d8dbb4f2e000a05c8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet fb6662cff60c4865acd527a45a257580 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 7c377df55c8049f6bd13fe8f53f52b28 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
f4c17323c7214a8d8dbb4f2e000a05c8 (127.21.184.193:43161): RUNNING [LEADER]
5264f57db51340f9900ee8ceb7998983 (127.21.184.194:45825): RUNNING
4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): TS unavailable
All reported replicas are:
A = f4c17323c7214a8d8dbb4f2e000a05c8
B = 5264f57db51340f9900ee8ceb7998983
C = 4cbab65af5824618b363b5a8b389be33
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 03:01:20.664682 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 03:01:21.244472 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260811 03:01:21.549803 22918 maintenance_manager.cc:419] P 37817f3438924819a8b313e319f2be3c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371
W20260811 03:01:21.553144 22917 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-task8YxRLs/test-tmp/all_types-itest.0.AllTypesItest_4.TestAllKeyTypes.1786417156762963-22243-0/minicluster-data/master-0/data/data (484970496 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 03:01:21.554088 22917 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 37817f3438924819a8b313e319f2be3c: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 03:01:21.554198 22917 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fa0ef6889ad google::LogMessage::Fail() at ??:0
@ 0x7fa0ef68b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0ef6884de google::LogMessage::Flush() at ??:0
@ 0x7fa0ef689a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0fea5de7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fa0f2176090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fa0f2178e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fa0f2178ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa0f22d4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa0f22d791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa0f22d7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa0f22b2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa0f558c6db start_thread at ??:0
@ 0x7fa0ee52261f clone at ??:0
W20260811 03:01:21.854424 26909 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:01:21.669267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.184.254:33799 (local address 127.0.0.1:49420)
0811 03:01:21.670129 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0811 03:01:21.670154 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:01:21.670318 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:01:21.852621 (+182303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":774,"threads_started":1}
W20260811 03:01:22.195009 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 03:01:22.408061 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:01:22.503371 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:01:22.683933 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:22.859241 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 03:01:23.725486 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 03:01:25.143518 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:25.144685 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 03:01:25.348325 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 03:01:26.437677 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 03:01:27.475617 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 03:01:27.709438 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 03:01:28.124260 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:29.225863 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 03:01:29.953318 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 03:01:30.354404 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:01:30.355438 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 03:01:31.779429 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 03:01:32.610654 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 03:01:32.861446 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 03:01:33.294840 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:34.311043 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 03:01:35.044613 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 03:01:35.141032 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 03:01:35.522557 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:01:36.945685 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 03:01:37.473367 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 03:01:37.780143 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 03:01:38.334071 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:39.555763 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 03:01:39.982324 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 03:01:40.260907 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 03:01:40.591109 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:42.137559 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 03:01:42.601866 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 03:01:42.700920 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 03:01:43.817580 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:44.980419 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 03:01:45.054306 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 03:01:45.279788 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 03:01:45.719396 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:01:47.479308 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 03:01:47.564559 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 03:01:47.681563 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 03:01:48.898422 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:49.810763 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 03:01:50.042714 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 03:01:50.158478 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 03:01:51.007756 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:01:52.180625 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 03:01:52.361872 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 03:01:52.543061 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 03:01:54.387020 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260811 03:01:54.388418 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 03:01:55.081175 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 03:01:55.379884 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 03:01:56.257673 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:01:56.959069 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 03:01:57.753599 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 03:01:57.860112 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 03:01:59.505972 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:01:59.507496 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 03:02:00.405114 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 03:02:00.451397 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 03:02:01.285530 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:02:02.275676 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 03:02:02.694545 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 03:02:02.983218 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 03:02:04.668156 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:04.669682 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 03:02:05.277454 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 03:02:05.365362 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 03:02:06.305001 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:02:07.285780 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 03:02:07.806573 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 03:02:07.869448 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 03:02:09.934816 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:09.936110 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 03:02:10.302726 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 03:02:10.528198 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 03:02:11.329350 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:02:12.622151 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 03:02:12.737994 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 03:02:13.023063 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 03:02:15.169368 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:15.170915 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 03:02:15.322629 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 03:02:15.783756 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 03:02:16.607163 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:17.538086 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 03:02:17.755035 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 03:02:18.042500 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 03:02:19.631810 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 03:02:20.167169 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 03:02:20.641207 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:02:20.764374 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 03:02:21.732632 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:02:21.862222 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:02:22.155882 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:02:22.497390 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (60 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:02:22.596812 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (60 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:02:22.670303 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 03:02:23.420557 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 03:02:24.234342 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 03:02:25.347247 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 03:02:25.774339 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:25.973059 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 03:02:26.391201 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 03:02:26.778465 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:27.847955 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 03:02:28.692476 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 03:02:28.910310 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 03:02:30.194886 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 03:02:31.155018 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:02:31.292208 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 03:02:31.403479 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 03:02:31.897542 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:02:32.788959 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 03:02:33.865950 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 03:02:33.963922 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 03:02:35.329910 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 03:02:36.238111 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:36.287365 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 03:02:36.559147 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 03:02:36.955758 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:37.585142 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 03:02:38.622730 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 03:02:38.843811 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 03:02:40.234103 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 03:02:41.129294 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 03:02:41.500551 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 03:02:41.563400 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:02:41.956362 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:02:42.606477 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 03:02:43.345880 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 03:02:43.990738 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 03:02:44.933995 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 03:02:45.955323 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 03:02:46.555037 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 03:02:46.740336 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:02:46.998134 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:47.345198 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 03:02:48.266265 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 03:02:48.811971 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 03:02:49.537298 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 03:02:50.839651 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 03:02:51.234356 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 03:02:52.109737 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:02:52.111179 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 03:02:52.250631 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:53.151245 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 03:02:53.697000 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 03:02:54.912299 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 03:02:55.886948 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 03:02:56.297761 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 03:02:57.284152 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:02:57.285573 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 03:02:57.511663 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:02:58.543326 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 03:02:58.551990 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 03:02:59.733249 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 03:03:01.272469 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 03:03:01.320070 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 03:03:01.988189 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 03:03:02.411087 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:03:02.602131 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:03.432124 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 03:03:03.843097 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 03:03:04.490127 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 03:03:05.824059 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 03:03:06.492698 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 03:03:07.131040 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 03:03:07.644061 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:07.684741 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:08.454382 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 03:03:09.216611 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 03:03:09.519192 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 03:03:10.786057 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 03:03:11.693854 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 03:03:12.249229 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 03:03:12.702867 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:03:13.167321 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:03:13.230604 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 03:03:14.234367 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 03:03:14.705297 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 03:03:15.772910 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 03:03:16.738584 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 03:03:17.259816 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 03:03:17.817413 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:18.297981 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 03:03:18.477418 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:19.160036 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 03:03:19.902390 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 03:03:20.892871 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 03:03:21.683497 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 03:03:22.481438 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 03:03:22.585172 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (120 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:03:22.683988 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (120 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:03:23.066311 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:03:23.489424 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:03:23.628564 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:23.660812 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 03:03:24.126367 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 03:03:24.855474 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 03:03:25.812382 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 03:03:26.403520 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 03:03:27.467278 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 03:03:28.173766 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:03:28.628083 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 03:03:28.974962 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:03:29.039230 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 03:03:29.991976 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 03:03:31.324859 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 03:03:31.775794 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 03:03:32.412093 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 03:03:33.288546 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:33.902369 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 03:03:33.991044 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:34.098591 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 03:03:35.048345 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 03:03:36.456984 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 03:03:36.492419 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 03:03:37.812003 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 03:03:38.334244 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:38.725576 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 03:03:39.073252 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 03:03:39.349269 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:40.459049 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 03:03:40.801487 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 03:03:41.595162 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 03:03:42.961066 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 03:03:43.462031 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:03:43.463255 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 03:03:44.274127 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 03:03:44.439864 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:45.387460 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 03:03:45.861510 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 03:03:46.712783 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 03:03:48.049706 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 03:03:48.393230 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 03:03:48.822010 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:03:49.283860 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 03:03:49.788686 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:03:50.373216 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 03:03:51.080616 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 03:03:51.822585 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 03:03:52.811949 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 03:03:53.463523 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 03:03:53.857697 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:03:54.270074 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 03:03:54.961884 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:03:55.405959 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 03:03:55.961982 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 03:03:56.856984 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 03:03:57.962177 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 03:03:58.691700 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 03:03:59.210282 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:03:59.211428 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 03:04:00.231736 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:04:00.232791 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 03:04:01.135675 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 03:04:01.508252 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 03:04:02.520358 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 03:04:03.809546 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 03:04:04.201609 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 03:04:04.426484 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:04:05.258767 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:05.260505 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 03:04:06.507220 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 03:04:06.508742 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 03:04:07.881255 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 03:04:08.939347 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 03:04:08.940254 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 03:04:09.504864 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:04:10.285352 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:10.286525 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 03:04:11.287753 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 03:04:11.563488 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 03:04:12.776623 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 03:04:13.739758 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 03:04:13.994613 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 03:04:14.563793 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:04:15.649457 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:15.651373 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 03:04:16.244062 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 03:04:16.428711 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 03:04:18.083659 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 03:04:18.776980 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 03:04:18.894235 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 03:04:19.765543 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:04:20.671712 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:20.673000 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 03:04:21.465608 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 03:04:21.466562 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260811 03:04:22.674245 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (180 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:04:22.771817 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (180 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:04:23.384279 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 03:04:24.027344 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 03:04:24.117679 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:04:24.783696 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:04:25.038224 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:04:25.977900 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:25.979310 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20260811 03:04:26.159901 27350 consensus_queue.cc:579] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Leader has been unable to successfully communicate with peer 4cbab65af5824618b363b5a8b389be33 for more than 300 seconds (300.036s)
I20260811 03:04:26.185994 27350 consensus_queue.cc:579] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 [LEADER]: Leader has been unable to successfully communicate with peer 4cbab65af5824618b363b5a8b389be33 for more than 300 seconds (300.224s)
I20260811 03:04:26.552057 27370 consensus_queue.cc:579] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 [LEADER]: Leader has been unable to successfully communicate with peer 4cbab65af5824618b363b5a8b389be33 for more than 300 seconds (300.486s)
W20260811 03:04:26.645257 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 03:04:26.737702 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 03:04:28.280054 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260811 03:04:29.074431 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 03:04:29.454838 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 03:04:30.204465 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:04:30.492458 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 03:04:30.979597 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:04:31.600535 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 03:04:31.754271 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 03:04:32.807421 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 03:04:34.301488 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 03:04:34.328905 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 03:04:35.212327 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 03:04:35.414311 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:04:36.229825 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:04:36.675773 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 03:04:37.121371 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 03:04:37.680347 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 03:04:39.451143 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 03:04:39.573369 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 03:04:40.126724 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 03:04:40.489056 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:04:41.691115 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:04:42.034178 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 03:04:42.076933 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 03:04:42.649928 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 03:04:44.499017 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 03:04:44.671118 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 03:04:45.020579 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 03:04:45.500591 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:04:47.021677 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 03:04:47.121965 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:04:47.179626 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 03:04:47.734478 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 03:04:49.784345 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 03:04:49.828222 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 03:04:50.405759 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 03:04:50.909502 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:04:52.310765 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 03:04:52.348791 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:52.570815 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 03:04:52.824841 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 03:04:54.922647 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 03:04:55.190331 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 03:04:55.368475 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 03:04:56.163306 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:04:57.158478 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 03:04:57.369252 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:04:57.807381 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 03:04:57.971930 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 03:04:59.551796 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 03:05:00.104871 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 03:05:00.183485 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 03:05:01.469576 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:05:02.075847 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 03:05:02.498621 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 03:05:02.513033 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:03.134662 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 03:05:04.595589 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 03:05:04.943594 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 03:05:05.387327 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 03:05:06.530054 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:05:07.086056 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 03:05:07.461545 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 03:05:07.728138 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:05:07.729164 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 03:05:09.454778 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 03:05:09.980007 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 03:05:10.555786 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 03:05:11.553210 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:05:11.698179 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 03:05:12.379482 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 03:05:12.916051 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:12.917003 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 03:05:14.053679 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 03:05:14.796337 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 03:05:15.380425 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 03:05:16.552031 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 03:05:16.725785 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:05:17.272960 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 03:05:17.872017 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 03:05:18.482678 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:05:18.672869 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 03:05:19.702278 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 03:05:20.583011 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 03:05:20.992130 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 03:05:21.753235 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:05:22.344003 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 03:05:22.897343 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 03:05:23.272668 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (242 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:05:23.433418 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 03:05:23.639083 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (241 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:05:23.800709 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:05:25.002789 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 03:05:25.521589 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 03:05:25.993271 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:05:26.089720 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:05:26.897455 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:05:27.478197 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 03:05:28.054132 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 03:05:28.391160 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 03:05:29.270006 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:29.976759 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 03:05:30.849123 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 03:05:31.008526 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 03:05:31.931751 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:05:32.460801 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 03:05:33.280593 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 03:05:33.442250 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 03:05:34.522464 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:34.939447 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 03:05:35.928033 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 03:05:36.281384 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 03:05:36.985877 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:05:37.725785 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 03:05:38.254951 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 03:05:38.746515 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 03:05:39.833149 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:40.118561 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 03:05:40.840797 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 03:05:41.518591 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 03:05:42.000033 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:05:42.691411 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 03:05:43.678689 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 03:05:44.131552 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 03:05:45.063411 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 03:05:45.247987 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:46.346664 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 03:05:46.490511 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 03:05:47.219560 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:05:47.299075 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 03:05:48.634791 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 03:05:48.792510 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 03:05:49.651489 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 03:05:50.466737 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:05:51.388151 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 03:05:51.630020 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 03:05:51.943276 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 03:05:52.316215 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:05:53.894047 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 03:05:53.975622 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 03:05:54.535212 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 03:05:55.599670 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:05:56.332938 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 03:05:56.541805 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 03:05:57.338187 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:05:57.379069 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 03:05:58.827271 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 03:05:59.051126 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 03:05:59.744262 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 03:06:00.725569 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:06:01.214293 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 03:06:01.572619 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 03:06:02.037101 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 03:06:02.535444 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:06:03.860610 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 03:06:04.329978 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 03:06:04.613547 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 03:06:06.229105 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:06.444572 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 03:06:06.816947 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 03:06:07.062111 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 03:06:07.585281 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:06:08.588148 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 03:06:09.296995 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 03:06:09.320106 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 03:06:11.180646 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 03:06:11.240002 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:11.627802 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 03:06:11.860774 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 03:06:12.749572 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:06:13.657895 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 03:06:13.830401 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 03:06:14.166076 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 03:06:16.194787 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 03:06:16.263262 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:16.313089 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 03:06:16.674507 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 03:06:18.042986 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:06:18.514272 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 03:06:18.629268 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 03:06:19.158681 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 03:06:21.029121 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 03:06:21.056118 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 03:06:21.277721 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:21.876504 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 03:06:23.132241 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:06:23.362107 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (302 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:06:23.636543 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 03:06:23.726531 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (301 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:06:23.736864 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 03:06:24.222777 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 03:06:26.274893 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 03:06:26.305297 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:26.321957 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 03:06:26.841048 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:06:27.414084 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:06:28.163715 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:06:28.540139 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 03:06:28.928753 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 03:06:29.293871 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 03:06:30.837544 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 03:06:31.453666 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:31.569998 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 03:06:32.070269 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 03:06:33.134229 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 03:06:33.549172 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:06:33.996671 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 03:06:34.795020 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 03:06:35.592866 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 03:06:36.531284 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 03:06:36.602823 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:37.018194 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 03:06:37.981007 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 03:06:38.859192 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:06:38.860345 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 03:06:39.527181 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 03:06:40.361657 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 03:06:41.231966 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 03:06:41.911778 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:06:41.912887 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 03:06:42.804461 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 03:06:43.616318 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 03:06:44.071523 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:06:44.326722 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 03:06:45.159328 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 03:06:46.346701 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 03:06:46.803303 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 03:06:47.267980 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:06:47.548274 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 03:06:49.156550 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:06:49.157666 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 03:06:49.250000 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 03:06:50.213899 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 03:06:51.665868 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 03:06:51.695312 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 03:06:52.290517 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:06:52.845186 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 03:06:54.014554 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 03:06:54.116782 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 03:06:54.217720 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:06:55.260025 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 03:06:56.465039 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 03:06:56.645414 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 03:06:57.529762 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:06:57.920171 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 03:06:59.156028 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 03:06:59.237197 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:06:59.238312 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 03:07:00.566671 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 03:07:01.440737 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 03:07:01.760176 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 03:07:02.540083 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:03.108320 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 03:07:03.963603 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 03:07:04.087426 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 03:07:04.610116 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:07:05.755039 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 03:07:06.406690 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 03:07:06.608296 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 03:07:08.050791 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:07:08.316948 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 03:07:09.046188 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 03:07:09.164428 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 03:07:09.770403 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:07:10.751730 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 03:07:11.312736 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 03:07:11.634477 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 03:07:13.046095 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 03:07:13.375396 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:07:13.954082 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 03:07:13.979099 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 03:07:14.998386 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:07:15.540941 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 03:07:16.593397 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 03:07:16.602717 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 03:07:17.794981 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 03:07:18.607604 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:19.090530 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 03:07:19.406172 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 03:07:20.406800 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:07:20.407848 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 03:07:21.750325 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 03:07:22.077379 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 03:07:23.157395 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 03:07:23.448931 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (362 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:07:23.816462 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (361 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:07:23.885850 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:24.400779 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 03:07:24.616925 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 03:07:25.473709 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:07:25.881079 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 03:07:26.837764 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 03:07:27.344700 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 03:07:28.459165 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:07:28.688364 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:07:28.971793 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:29.598963 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 03:07:30.000281 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 03:07:30.511482 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:07:30.893379 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 03:07:32.164124 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 03:07:32.450598 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 03:07:33.265566 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 03:07:33.978533 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:34.543767 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 03:07:35.252485 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 03:07:35.702427 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:07:35.703565 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 03:07:36.907668 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 03:07:37.744449 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 03:07:38.170981 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 03:07:39.115703 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:07:39.690827 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 03:07:40.121855 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 03:07:40.838536 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:07:40.839740 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 03:07:42.016959 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 03:07:42.629609 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 03:07:43.417971 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 03:07:44.481853 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:07:44.483201 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 03:07:44.974025 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 03:07:45.866180 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:07:45.931898 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 03:07:46.803833 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 03:07:47.552983 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 03:07:48.617161 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 03:07:49.255764 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 03:07:49.724095 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 03:07:49.837348 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:07:50.919332 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:07:51.487285 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 03:07:51.608510 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 03:07:52.310019 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 03:07:54.254670 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 03:07:54.277516 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 03:07:55.069070 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 03:07:55.322036 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:07:56.008867 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:07:56.755955 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 03:07:56.784055 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 03:07:57.411684 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 03:07:59.276892 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 03:07:59.378790 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 03:07:59.782060 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 03:08:00.456895 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:08:01.195150 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:01.648121 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 03:08:01.693140 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 03:08:02.224826 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 03:08:03.935824 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 03:08:04.156942 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 03:08:04.929019 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 03:08:05.850903 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:06.411559 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:06.412806 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 03:08:06.769075 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 03:08:07.321699 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 03:08:08.984871 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 03:08:09.378665 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 03:08:09.927721 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 03:08:10.979249 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:08:11.360414 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 03:08:11.813123 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:08:11.838691 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 03:08:12.523540 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 03:08:13.896929 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 03:08:14.262555 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 03:08:14.953079 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 03:08:16.145785 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:16.316963 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 03:08:16.713960 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 03:08:16.965543 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:17.351236 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 03:08:18.620530 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 03:08:19.318928 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 03:08:19.721033 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 03:08:20.999727 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 03:08:21.399087 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:08:21.966694 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 03:08:22.000758 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:22.249594 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 03:08:23.414633 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 03:08:23.534099 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (422 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:08:23.905972 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (421 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:08:24.323935 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 03:08:24.493062 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 03:08:25.968212 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 03:08:26.592589 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:26.593729 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 03:08:26.731971 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 03:08:27.333256 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:08:28.512910 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 03:08:28.856029 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 03:08:29.323956 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:08:29.982868 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:08:30.734645 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 03:08:31.398686 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 03:08:31.907502 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:31.938844 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 03:08:32.446972 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:33.122939 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 03:08:33.934345 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 03:08:34.454105 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 03:08:35.739203 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 03:08:36.315994 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 03:08:37.181159 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 03:08:37.465348 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:37.485438 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:08:38.309993 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 03:08:38.985008 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 03:08:39.362583 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 03:08:40.499114 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 03:08:41.486193 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 03:08:41.922382 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 03:08:42.500312 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:42.824409 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:43.080672 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 03:08:43.772186 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 03:08:44.530732 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 03:08:45.437654 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 03:08:46.163897 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 03:08:46.957333 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 03:08:47.554195 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:47.963835 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:08:48.174538 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 03:08:48.496557 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 03:08:49.247437 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 03:08:50.883623 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 03:08:51.186291 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 03:08:51.639683 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 03:08:52.936321 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:53.113257 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:08:53.312906 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 03:08:53.619557 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 03:08:54.004536 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 03:08:55.582556 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 03:08:56.400548 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 03:08:56.421480 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 03:08:58.081727 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:08:58.178449 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 03:08:58.413726 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:08:59.035822 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 03:08:59.062044 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 03:09:00.482074 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 03:09:01.539629 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 03:09:01.860605 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 03:09:02.749832 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 03:09:03.156116 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:03.636757 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:04.043540 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 03:09:04.463691 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 03:09:05.081203 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 03:09:06.522864 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 03:09:06.996713 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 03:09:07.686383 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 03:09:08.558419 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:08.694482 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:09.109181 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 03:09:09.757242 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 03:09:09.850102 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 03:09:11.494766 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 03:09:12.250051 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 03:09:12.300470 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 03:09:13.969913 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:09:13.971481 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:09:13.972462 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 03:09:14.488708 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 03:09:15.041857 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 03:09:16.264793 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 03:09:16.934969 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 03:09:17.370868 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 03:09:18.987982 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:18.989321 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 03:09:19.054729 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:19.535316 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 03:09:19.777432 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 03:09:21.189677 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 03:09:22.222255 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 03:09:22.229869 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 03:09:23.620024 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (482 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:09:23.855834 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 03:09:23.995937 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (481 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:09:24.260600 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:09:24.354480 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:09:24.693781 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 03:09:24.867581 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 03:09:26.126178 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 03:09:27.278584 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 03:09:27.597783 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 03:09:28.516407 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 03:09:29.377781 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:09:29.540539 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:09:29.786010 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 03:09:29.956938 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 03:09:31.174870 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:09:31.308341 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:09:31.996749 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 03:09:32.628089 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 03:09:33.563658 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 03:09:34.542304 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:34.543352 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 03:09:34.934738 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:09:35.084272 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 03:09:36.040968 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 03:09:37.049746 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 03:09:37.530275 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 03:09:38.539494 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 03:09:39.550184 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:39.551220 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 03:09:39.875967 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 03:09:39.970549 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:41.048077 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 03:09:41.999938 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 03:09:42.107725 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 03:09:43.546088 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 03:09:44.074312 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 03:09:44.604198 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:44.857385 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 03:09:45.028165 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:45.995508 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 03:09:46.524094 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 03:09:47.188879 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 03:09:48.576269 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 03:09:49.117189 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 03:09:49.764216 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:49.765357 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 03:09:50.094934 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:51.322233 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 03:09:51.726763 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 03:09:52.162791 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 03:09:53.560510 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 03:09:54.102788 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 03:09:54.670768 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 03:09:55.127597 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:09:55.134686 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:09:56.090375 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 03:09:56.737920 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 03:09:56.958173 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 03:09:58.508359 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 03:09:58.954618 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 03:09:59.458531 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 03:10:00.163223 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:00.330643 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:10:01.282279 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 03:10:01.319734 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 03:10:01.923166 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 03:10:03.855765 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 03:10:03.889453 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 03:10:04.329855 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 03:10:05.231292 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:05.435817 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:10:06.297935 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 03:10:06.456569 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 03:10:07.038537 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 03:10:08.565605 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 03:10:08.738092 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 03:10:09.449520 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 03:10:10.301600 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:10.585714 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:11.004987 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 03:10:11.131886 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 03:10:11.931272 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 03:10:13.345244 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 03:10:13.503804 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 03:10:14.552887 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 03:10:15.644506 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:15.672788 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260811 03:10:15.674227 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 03:10:16.005884 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 03:10:17.078168 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 03:10:18.186851 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 03:10:18.589973 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 03:10:19.839715 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 03:10:20.548543 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 03:10:20.831003 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:10:20.846309 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 03:10:21.007165 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:10:22.474104 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 03:10:22.960314 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 03:10:23.457952 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 03:10:23.704799 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (542 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:10:24.084573 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (541 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:10:24.912438 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 03:10:25.348573 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 03:10:25.804771 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 03:10:26.053830 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:26.465770 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:10:27.590052 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 03:10:27.716596 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 03:10:28.046561 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 03:10:29.736507 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 03:10:30.354604 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 03:10:30.536276 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 03:10:31.236595 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:32.018764 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20260811 03:10:32.542421 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:10:32.621227 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:10:32.858561 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 03:10:32.969699 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 03:10:34.826081 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 03:10:35.175633 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 03:10:35.410627 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 03:10:36.396996 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:37.056067 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:37.494915 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 03:10:37.540370 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 03:10:37.757274 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 03:10:39.858418 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 03:10:39.911476 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 03:10:39.961680 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 03:10:41.521479 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:10:42.282539 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:10:42.283591 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 03:10:42.358815 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 03:10:42.543496 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 03:10:44.801932 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 03:10:44.862428 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 03:10:44.994342 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 03:10:46.644783 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:10:47.127857 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 03:10:47.264919 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 03:10:47.478804 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:47.480054 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 03:10:49.477082 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 03:10:49.647629 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 03:10:50.128638 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 03:10:51.950037 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:10:51.951169 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 03:10:52.026058 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 03:10:52.703953 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:10:52.705022 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 03:10:54.445240 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 03:10:54.592832 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 03:10:55.402809 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 03:10:56.970957 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:10:56.971968 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 03:10:57.099944 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 03:10:57.599248 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 03:10:58.104902 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:10:59.510964 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 03:10:59.568059 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 03:11:00.365257 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 03:11:01.898577 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 03:11:02.122834 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:11:02.124132 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 03:11:02.976928 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 03:11:03.500418 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:04.285567 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 03:11:04.603982 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 03:11:05.523645 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 03:11:06.825027 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 03:11:06.936800 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 03:11:07.190718 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:11:07.909457 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 03:11:08.501739 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:09.271682 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 03:11:09.414830 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 03:11:10.405181 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 03:11:11.642724 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 03:11:11.788923 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 03:11:12.230041 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:12.656584 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 03:11:13.678377 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:11:14.157099 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 03:11:14.213202 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 03:11:15.429461 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 03:11:16.335579 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 03:11:16.963898 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 03:11:17.343061 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:11:18.094570 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 03:11:18.784730 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 03:11:19.124619 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:19.571102 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 03:11:20.703963 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 03:11:21.246019 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 03:11:22.084362 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 03:11:22.376734 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:23.281930 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 03:11:23.792002 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (602 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:11:23.916774 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 03:11:24.172675 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (601 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:11:24.221236 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:24.482270 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 03:11:25.581259 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 03:11:26.810173 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 03:11:27.053764 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 03:11:27.666682 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:27.912478 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 03:11:29.527590 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 03:11:29.562273 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 03:11:29.615815 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:11:30.704952 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 03:11:31.873513 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 03:11:32.293434 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 03:11:32.719102 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:33.161381 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:11:33.936054 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:11:34.265479 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 03:11:34.700990 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 03:11:34.772967 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:35.719404 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 03:11:36.888392 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 03:11:37.027099 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 03:11:37.926414 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 03:11:38.036629 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:11:39.335337 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 03:11:39.814424 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 03:11:39.833321 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:11:40.319556 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 03:11:41.963961 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 03:11:41.969686 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 03:11:42.853386 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 03:11:43.070588 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:44.396672 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 03:11:44.612455 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 03:11:45.250746 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:11:45.251823 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 03:11:46.384871 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 03:11:47.128789 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 03:11:47.881498 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 03:11:48.188459 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:11:48.732636 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 03:11:49.673312 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 03:11:50.282843 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:50.283876 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 03:11:51.292618 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 03:11:52.095448 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 03:11:52.802655 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 03:11:53.332249 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:11:53.996773 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 03:11:54.488021 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 03:11:55.441593 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:11:55.442911 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 03:11:56.438460 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 03:11:57.104065 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 03:11:57.780809 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 03:11:58.394344 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:11:59.051788 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 03:11:59.454532 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 03:12:00.443881 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:00.444998 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 03:12:01.477527 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 03:12:01.653005 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 03:12:02.876272 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 03:12:03.703241 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:12:03.781529 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 03:12:04.142700 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 03:12:05.376502 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 03:12:05.815514 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:12:06.302394 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 03:12:06.629236 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 03:12:07.848183 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 03:12:08.879833 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:12:08.880824 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 03:12:08.993908 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 03:12:10.670496 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 03:12:11.205694 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:11.213197 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 03:12:11.284433 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 03:12:13.220510 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 03:12:13.684322 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 03:12:13.795083 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 03:12:14.282291 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:12:15.846334 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 03:12:16.226006 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 03:12:16.241933 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 03:12:16.280661 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:18.189314 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 03:12:18.715389 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 03:12:18.771549 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 03:12:19.319528 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:12:20.724426 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 03:12:21.202692 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 03:12:21.289673 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:21.516464 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 03:12:23.267890 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 03:12:23.839942 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 03:12:23.880129 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (662 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:12:24.008255 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 03:12:24.262876 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (661 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:12:24.409828 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:12:25.869431 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 03:12:26.287207 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 03:12:26.422758 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 03:12:26.440303 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:28.616281 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 03:12:28.877861 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 03:12:29.081003 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 03:12:29.511489 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:12:31.193202 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 03:12:31.202415 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 03:12:31.664278 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 03:12:31.713470 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:33.523586 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 03:12:33.710557 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 03:12:33.776016 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 03:12:34.568460 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:12:35.235333 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:12:35.867869 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 03:12:35.972718 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 03:12:36.175240 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 03:12:37.069896 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:12:38.218405 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 03:12:38.533149 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 03:12:38.652046 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 03:12:39.659264 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:12:40.771324 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 03:12:40.981647 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 03:12:41.167508 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 03:12:42.181560 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:42.984406 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 03:12:43.698341 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 03:12:43.807279 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 03:12:45.093611 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:12:45.735045 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 03:12:46.149128 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 03:12:46.441680 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 03:12:47.347518 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:48.590445 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 03:12:48.791216 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 03:12:48.791615 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 03:12:50.122380 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:12:51.213269 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 03:12:51.294530 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 03:12:51.394089 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 03:12:52.472641 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:53.710009 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 03:12:53.723829 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 03:12:53.768440 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 03:12:55.209714 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:12:56.196782 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 03:12:56.270087 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 03:12:56.462387 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 03:12:57.486846 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:12:58.740742 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 03:12:58.845741 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 03:12:59.104987 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 03:13:00.321545 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:13:01.085515 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 03:13:01.195282 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 03:13:01.465317 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 03:13:02.971028 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:13:03.533459 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 03:13:03.622783 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 03:13:04.087347 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 03:13:05.668713 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:13:06.142225 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 03:13:06.267925 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 03:13:06.864233 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 03:13:08.074725 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260811 03:13:08.496779 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 03:13:08.815382 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 03:13:09.572252 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 03:13:10.720945 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:13:10.721956 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 03:13:11.366478 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 03:13:12.025223 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 03:13:13.398300 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 03:13:13.406878 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:13:13.607143 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 03:13:14.536322 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 03:13:15.887995 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:13:15.888952 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 03:13:16.147809 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 03:13:17.131503 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 03:13:18.216563 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 03:13:18.304323 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 03:13:18.457394 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:13:19.390039 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 03:13:20.581593 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 03:13:20.632344 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 03:13:20.991220 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20260811 03:13:21.970041 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 03:13:22.962495 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 03:13:23.160476 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 03:13:23.771919 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:13:23.966090 23137 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (722 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:13:24.352094 23281 heartbeater.cc:644] Failed to heartbeat to 127.21.184.254:33799 (721 consecutive failures): Network error: Failed to ping master at 127.21.184.254:33799: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:13:24.392041 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 03:13:25.614682 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 03:13:25.878238 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 03:13:26.189924 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:13:26.966992 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 03:13:28.065758 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 03:13:28.528402 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 03:13:28.891618 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:13:29.378706 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 03:13:30.718639 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 03:13:30.844193 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 03:13:31.230383 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:13:31.793990 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 03:13:33.264479 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 03:13:33.326949 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 03:13:34.116902 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:13:34.118242 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 03:13:35.905375 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 03:13:35.937527 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 03:13:36.319561 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.21.184.254:33799) | 127.21.184.254:33799 | UNAVAILABLE
Error from 127.21.184.254:33799: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
FAILED
I20260811 03:13:36.552901 22243 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.21.184.254:33799, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.21.184.254:33799: connect: Connection refused (error 111)
W20260811 03:13:36.871371 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 03:13:38.378788 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 03:13:38.634961 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 03:13:39.322327 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:13:39.323405 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 03:13:40.743574 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 03:13:41.402036 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:13:41.403230 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 03:13:41.829435 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 03:13:43.462934 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 03:13:44.039034 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 03:13:44.186914 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 03:13:44.787380 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:13:46.218286 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 03:13:46.659344 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 03:13:46.724642 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:13:46.816066 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
[New LWP 22246]
[New LWP 22247]
[New LWP 22891]
[New LWP 23428]
[New LWP 23429]
[New LWP 23430]
[New LWP 23431]
[New LWP 28686]
[New LWP 28687]
[New LWP 28688]
[New LWP 28689]
[New LWP 28692]
[New LWP 28693]
[New LWP 28694]
[New LWP 28695]
[New LWP 28696]
W20260811 03:13:48.757205 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 03:13:49.172739 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 03:13:49.354169 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 03:13:49.904138 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:13:51.350963 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 03:13:51.724177 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 03:13:51.797494 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:13:52.017438 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 03:13:53.865789 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 03:13:54.486243 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 03:13:54.542735 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 03:13:55.072430 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:13:56.482772 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 03:13:56.895844 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20260811 03:13:57.028891 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 03:13:57.054728 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 03:13:58.957549 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 03:13:59.455262 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 03:13:59.520004 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 03:14:00.506989 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:14:01.487809 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 03:14:01.943284 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 03:14:02.020023 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:14:02.021884 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 03:14:03.857928 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 03:14:04.215344 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 03:14:04.381887 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 03:14:05.775959 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:14:06.282732 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 03:14:06.778681 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 03:14:07.022841 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20260811 03:14:07.024099 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 03:14:08.675855 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260811 03:14:09.431272 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 03:14:09.608069 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 03:14:10.867544 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260811 03:14:11.217810 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 03:14:11.849341 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 03:14:12.152277 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20260811 03:14:12.153450 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
0x00007f58733ffad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f586b900d40 (LWP 22243) "all_types-itest" 0x00007f58733ffad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f58674bd700 (LWP 22246) "all_types-itest" 0x00007f587340432a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f5865ef4700 (LWP 22247) "kernel-watcher-" 0x00007f58733fffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f58634e8700 (LWP 22891) "rpc reactor-228" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f5863ce9700 (LWP 23428) "rpc reactor-234" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f58644ea700 (LWP 23429) "rpc reactor-234" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f5864ef2700 (LWP 23430) "rpc reactor-234" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f58656f3700 (LWP 23431) "rpc reactor-234" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f58619c1700 (LWP 28686) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f58611c0700 (LWP 28687) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f585d0c4700 (LWP 28688) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f585e0c6700 (LWP 28689) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f58609bf700 (LWP 28692) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f585f8c9700 (LWP 28693) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f585f0c8700 (LWP 28694) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f585e8c7700 (LWP 28695) "rpc reactor-286" 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f585d8c5700 (LWP 28696) "client-negotiat" 0x00007f58733fffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f585d8c5700 (LWP 28696)):
#0 0x00007f58733fffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f58705ea8d9 in kudu::ConditionVariable::WaitFor (this=0x7f58733fffb9 <pthread_cond_timedwait@@GLIBC_2.3.2+649>, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007f587089ae0a in kudu::ThreadPool::DispatchThread (this=0x616000150f80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f587089e91c in kudu::ThreadPool::CreateThread()::$_2::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
W20260811 03:14:14.124861 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 03:14:14.254807 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
#4 0x00007f587089e799 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#5 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100019e3e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#6 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f585e8c7700 (LWP 28695)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x619000263e80, timeout=0.099117313511669636) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
W20260811 03:14:14.628234 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#2 0x00007f586e7c6228 in ev_run (loop=0x619000263e80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005a288, flags=305664, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005a280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110000e11a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f585f0c8700 (LWP 28694)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x619000263480, timeout=0.099035978317260742) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x619000263480, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005a488, flags=1284992, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005a480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110000e16a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f585f8c9700 (LWP 28693)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x619000263980, timeout=0.099090566392987967) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x619000263980, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005a888, flags=306560, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005a880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110000e17e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f58609bf700 (LWP 28692)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x61900021ac80, timeout=0.088783310959115624) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x61900021ac80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005a688, flags=1337856, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005a680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110000e1928) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f585e0c6700 (LWP 28689)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x61900135f180, timeout=0.099384637316688895) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x61900135f180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005ae88, flags=1335168, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005ae80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x611000130028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f585d0c4700 (LWP 28688)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x61900135ec80, timeout=0.099353088531643152) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x61900135ec80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005b088, flags=1283200, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005b080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x611000130168) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f58611c0700 (LWP 28687)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x61900135e780, timeout=0.09935992956161499) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x61900135e780, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005b288, flags=5945984, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005b280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110001302a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f58619c1700 (LWP 28686)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x61900135e280, timeout=0.099329214310273528) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x61900135e280, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400005b488, flags=6290048, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400005b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x6110001303e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f58656f3700 (LWP 23431)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x6190000cd080, timeout=0.099332668352872133) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x6190000cd080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400000a888, flags=49408, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400000a880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100008c928) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f5864ef2700 (LWP 23430)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x6190000ccb80, timeout=0.099251392297446728) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x6190000ccb80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400000a688, flags=48512, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400000a680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100008c7e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f58644ea700 (LWP 23429)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x6190000cc680, timeout=0.099239368224516511) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x6190000cc680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400000a488, flags=47616, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400000a480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100008c6a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f5863ce9700 (LWP 23428)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x6190000cc180, timeout=0.099195165326818824) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x6190000cc180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x61400000a288, flags=46720, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x61400000a280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100008c568) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f58634e8700 (LWP 22891)):
#0 0x00007f586c6d3947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f586e7c2bda in epoll_poll (loop=0x6190000ca380, timeout=0.099625358125194907) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f586e7c6228 in ev_run (loop=0x6190000ca380, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f5876e5ee31 in ev::loop_ref::run (this=this@entry=0x614000009e88, flags=43136, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f5876e57326 in kudu::rpc::ReactorThread::RunThread (this=0x614000009e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f5876e5c3ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f5876e5c229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x61100009c1e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f5865ef4700 (LWP 22247)):
#0 0x00007f58733fffb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f58705ea50e in kudu::ConditionVariable::WaitUntil (this=0x612000009118, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
#2 0x00007f588443d02c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f58844354aa in kudu::CountDownLatch::WaitFor (this=0x612000009140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f587072045f in kudu::KernelStackWatchdog::RunThread (this=0x612000009040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f587072151c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f5870721399 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f5884df5bbd in std::function<void ()>::operator()() const (this=0x611000052228) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f5870879eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f58674bd700 (LWP 22246)):
#0 0x00007f587340432a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000039b4fb in __interceptor_waitpid (pid=<optimized out>, status=0x7f58674bb8b0, options=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f5870864796 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#3 0x00007f58708644c5 in kudu::Subprocess::Wait (this=0x7f58674bb8b0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#4 0x00007f58706fa2be in kudu::PstackWatcher::RunStackDump (argv=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#5 0x00007f58706f99a7 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f58706f8e96 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f58706f7417 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000005386f9 in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#9 0x0000000000538583 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#10 0x0000000000538503 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#11 0x00000000005384d9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#12 0x0000000000538499 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#13 0x000000000053839b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#14 0x00007f5877d40a4f in execute_native_thread_routine () from /tmp/dist-test-task8YxRLs/build/asan/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#15 0x00007f58733f96db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f586c6d361f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f586b900d40 (LWP 22243)):
#0 0x00007f58733ffad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f58705ea1e7 in kudu::ConditionVariable::Wait (this=0x60e00006ad70) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
#2 0x00007f58854ac1b9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260811 03:14:16.298342 23158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260811 03:14:16.604709 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260811 03:14:16.726666 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 03:14:16.863710 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#3 0x00007f5884ba2f4b in kudu::Synchronizer::Wait (this=0x7ffe46b17ec0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f587de43ef3 in kudu::client::KuduClient::Data::ConnectToCluster (this=0x615000bbc900, client=<optimized out>, deadline=..., creds_policy=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260811 03:14:17.367726 23012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111) [suppressed 20 similar messages]
#5 0x00007f587ddc3901 in kudu::client::KuduClientBuilder::Build (this=0x7ffe46b187c0, client=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
#6 0x00007f587e769eac in kudu::tools::CreateKuduClient (master_addresses=..., client=<optimized out>, can_see_all_replicas=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
#7 0x00007f5884462c15 in kudu::tools::RemoteKsckCluster::Connect (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:528
#8 0x00007f58843d687e in kudu::tools::Ksck::CheckClusterRunning (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f58843d883f in kudu::tools::Ksck::Run (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f58843dd13d in kudu::tools::Ksck::RunAndPrintResults (this=0x61900025f390) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
W20260811 03:14:19.080214 23012 consensus_peers.cc:487] T 7c377df55c8049f6bd13fe8f53f52b28 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#11 0x00007f5885454dd3 in kudu::ClusterVerifier::RunKsck (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f5885454556 in kudu::ClusterVerifier::CheckCluster (this=0x7ffe46b19be0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
W20260811 03:14:19.380651 23158 consensus_peers.cc:487] T bbc3e9de57f1419aa78b468e7d684a89 P 5264f57db51340f9900ee8ceb7998983 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
#13 0x0000000000494b38 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x60c0000e2780, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x000000000049382d in kudu::client::AllTypesItest_TestAllKeyTypes_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> > >::TestBody (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:676
W20260811 03:14:19.517352 23012 consensus_peers.cc:487] T fb6662cff60c4865acd527a45a257580 P f4c17323c7214a8d8dbb4f2e000a05c8 -> Peer 4cbab65af5824618b363b5a8b389be33 (127.21.184.195:35255): Couldn't send request to peer 4cbab65af5824618b363b5a8b389be33. Status: Network error: Client connection negotiation failed: client connection to 127.21.184.195:35255: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#15 0x00007f58710d3bfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f58710d9a28 "the test body", method=<optimized out>, object=0x60c0000e2780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x60c0000e2780, method=<optimized out>, location=location@entry=0x7f58710d9a28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f58710c85db in testing::Test::Run (this=0x60c0000e2780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x60c0000e2780) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f58710c8795 in testing::TestInfo::Run (this=0x611000045dc0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f58710c8cdf in testing::TestSuite::Run (this=0x611000045f00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x611000045f00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f58710c929f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f58710d413e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f58710db5a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x61600000b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x61600000b480, method=<optimized out>, location=location@entry=0x7f58710db5a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f58710c883d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x0000000000539482 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x0000000000537992 in main (argc=<optimized out>, argv=0x7ffe46b1a468) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20260811 03:14:20.014791 22246 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786418060 (unix time) try "date -d @1786418060" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000056e3) received by PID 22243 (TID 0x7f58674bd700) from PID 22243; stack trace: ***
@ 0x7f586f2c2a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f5873404980 (unknown) at ??:0
@ 0x7f586c5f0e87 gsignal at ??:0
@ 0x7f586c5f27f1 abort at ??:0
@ 0x7f586f2b49ad google::LogMessage::Fail() at ??:0
@ 0x7f586f2b788a google::LogMessage::SendToLog() at ??:0
@ 0x7f586f2b44de google::LogMessage::Flush() at ??:0
@ 0x7f586f2b5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x538891 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63
@ 0x538583 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
@ 0x538503 std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x5384d9 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x538499 std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x53839b std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7f5877d40a4f execute_native_thread_routine at ??:0
@ 0x7f58733f96db start_thread at ??:0
@ 0x7f586c6d361f clone at ??:0