Note: This is test shard 6 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/2, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> >
[ RUN ] AllTypesItest/2.TestTimestampPadding
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260807 18:15:27.232141 27665 test_util.cc:268] Using random seed: -47318979
2026-08-07T18:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T18:15:27Z Disabled control of system clock
I20260807 18:15:27.287206 27665 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
/tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.4.126:46561
--webserver_interface=127.27.4.126
--webserver_port=0
--builtin_ntp_servers=127.27.4.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.4.126:46561 with env {}
W20260807 18:15:27.644237 27675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:15:27.644537 27675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:15:27.644625 27675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:15:27.651871 27675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 18:15:27.651993 27675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:15:27.652038 27675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 18:15:27.652087 27675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 18:15:27.661377 27675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.4.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.4.126:46561
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.4.126:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.4.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:15:27.662765 27675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:15:27.664567 27675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:15:27.677083 27680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:27.677119 27681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:27.677017 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:15:27.677485 27675 server_base.cc:1034] running on GCE node
I20260807 18:15:27.678397 27675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:15:27.679738 27675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:15:27.681026 27675 hybrid_clock.cc:648] HybridClock initialized: now 1786126527681000 us; error 89 us; skew 500 ppm
I20260807 18:15:27.681525 27675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:15:27.684008 27675 webserver.cc:460] Webserver started at http://127.27.4.126:33053/ using document root <none> and password file <none>
I20260807 18:15:27.684723 27675 fs_manager.cc:359] Metadata directory not provided
I20260807 18:15:27.684877 27675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:15:27.685205 27675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:15:27.688052 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/instance:
uuid: "767d100e67004aa29f1b558de36175c0"
format_stamp: "Formatted at 2026-08-07 18:15:27 on dist-test-slave-bw88"
I20260807 18:15:27.688846 27675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal/instance:
uuid: "767d100e67004aa29f1b558de36175c0"
format_stamp: "Formatted at 2026-08-07 18:15:27 on dist-test-slave-bw88"
I20260807 18:15:27.694106 27675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 18:15:27.697777 27691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:27.699199 27675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 18:15:27.699468 27675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data,/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal
uuid: "767d100e67004aa29f1b558de36175c0"
format_stamp: "Formatted at 2026-08-07 18:15:27 on dist-test-slave-bw88"
I20260807 18:15:27.699700 27675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:15:27.713173 27675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:15:27.714110 27675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:15:27.714437 27675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:15:27.734297 27675 rpc_server.cc:307] RPC server started. Bound to: 127.27.4.126:46561
I20260807 18:15:27.734331 27756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.4.126:46561 every 8 connection(s)
I20260807 18:15:27.735842 27675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb
I20260807 18:15:27.740303 27757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:27.745007 27665 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu as pid 27675
I20260807 18:15:27.745359 27665 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/wal/instance
I20260807 18:15:27.753916 27757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: Bootstrap starting.
I20260807 18:15:27.757599 27757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:27.758751 27757 log.cc:824] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: Log is configured to *not* fsync() on all Append() calls
I20260807 18:15:27.762392 27757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: No bootstrap required, opened a new log
I20260807 18:15:27.773815 27757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } }
I20260807 18:15:27.774240 27757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:27.774374 27757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767d100e67004aa29f1b558de36175c0, State: Initialized, Role: FOLLOWER
I20260807 18:15:27.775171 27757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } }
I20260807 18:15:27.775552 27757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 18:15:27.775717 27757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 18:15:27.775956 27757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:27.778489 27757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } }
I20260807 18:15:27.779197 27757 leader_election.cc:302] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767d100e67004aa29f1b558de36175c0; no voters:
I20260807 18:15:27.780026 27757 leader_election.cc:288] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 18:15:27.780189 27764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:15:27.782326 27764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [term 1 LEADER]: Becoming Leader. State: Replica: 767d100e67004aa29f1b558de36175c0, State: Running, Role: LEADER
I20260807 18:15:27.783032 27764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } }
I20260807 18:15:27.783182 27757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 18:15:27.786773 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767d100e67004aa29f1b558de36175c0. Latest consensus state: current_term: 1 leader_uuid: "767d100e67004aa29f1b558de36175c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } } }
I20260807 18:15:27.787150 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767d100e67004aa29f1b558de36175c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767d100e67004aa29f1b558de36175c0" member_type: VOTER last_known_addr { host: "127.27.4.126" port: 46561 } } }
I20260807 18:15:27.787354 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:15:27.787468 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0 [sys.catalog]: This master's current role is: LEADER
I20260807 18:15:27.789872 27774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 18:15:27.795601 27774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 18:15:27.806730 27774 catalog_manager.cc:1347] Generated new cluster ID: 1e1bc3e5af984cc3b2659fe8d56feb5c
I20260807 18:15:27.806878 27774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 18:15:27.819186 27774 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 18:15:27.821084 27774 catalog_manager.cc:1504] Loading token signing keys...
I20260807 18:15:27.831204 27774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: Generated new TSK 0
I20260807 18:15:27.832492 27774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 18:15:27.848618 27665 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
/tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.4.65:0
--local_ip_for_outbound_sockets=127.27.4.65
--webserver_interface=127.27.4.65
--webserver_port=0
--tserver_master_addrs=127.27.4.126:46561
--builtin_ntp_servers=127.27.4.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260807 18:15:28.209581 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:15:28.209919 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:15:28.210042 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:15:28.217795 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:15:28.218019 27788 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260807 18:15:28.218106 27788 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260807 18:15:28.218223 27788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.4.65
I20260807 18:15:28.227916 27788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.4.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.4.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.4.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.27.4.126:46561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.4.65
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:15:28.229408 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:15:28.231393 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:15:28.246018 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:28.246037 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:28.246043 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:15:28.247232 27788 server_base.cc:1034] running on GCE node
I20260807 18:15:28.247927 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:15:28.248904 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:15:28.250162 27788 hybrid_clock.cc:648] HybridClock initialized: now 1786126528250083 us; error 57 us; skew 500 ppm
I20260807 18:15:28.250486 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:15:28.252836 27788 webserver.cc:460] Webserver started at http://127.27.4.65:38185/ using document root <none> and password file <none>
I20260807 18:15:28.253672 27788 fs_manager.cc:359] Metadata directory not provided
I20260807 18:15:28.253824 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:15:28.254161 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:15:28.256819 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/instance:
uuid: "b70fc93f83824bcca1b7252f86a22dc7"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.257643 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal/instance:
uuid: "b70fc93f83824bcca1b7252f86a22dc7"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.263017 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 18:15:28.266461 27804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:28.267778 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:15:28.267953 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal
uuid: "b70fc93f83824bcca1b7252f86a22dc7"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.268147 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:15:28.283315 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:15:28.284374 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:15:28.284695 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:15:28.285790 27788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:15:28.287334 27788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:15:28.287442 27788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:28.287608 27788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:15:28.287683 27788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:28.323962 27788 rpc_server.cc:307] RPC server started. Bound to: 127.27.4.65:35371
I20260807 18:15:28.324020 27949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.4.65:35371 every 8 connection(s)
I20260807 18:15:28.325415 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/info.pb
I20260807 18:15:28.331535 27665 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu as pid 27788
I20260807 18:15:28.331749 27665 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/wal/instance
I20260807 18:15:28.336016 27665 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
/tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.4.66:0
--local_ip_for_outbound_sockets=127.27.4.66
--webserver_interface=127.27.4.66
--webserver_port=0
--tserver_master_addrs=127.27.4.126:46561
--builtin_ntp_servers=127.27.4.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260807 18:15:28.342185 27950 heartbeater.cc:347] Connected to a master server at 127.27.4.126:46561
I20260807 18:15:28.342543 27950 heartbeater.cc:464] Registering TS with master...
I20260807 18:15:28.343423 27950 heartbeater.cc:511] Master 127.27.4.126:46561 requested a full tablet report, sending...
I20260807 18:15:28.345626 27710 ts_manager.cc:194] Registered new tserver with Master: b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
I20260807 18:15:28.347122 27710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.4.65:60923
W20260807 18:15:28.684568 27954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:15:28.684935 27954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:15:28.685087 27954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:15:28.692701 27954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:15:28.692902 27954 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260807 18:15:28.692992 27954 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260807 18:15:28.693158 27954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.4.66
I20260807 18:15:28.702432 27954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.4.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.4.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.4.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.27.4.126:46561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.4.66
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:15:28.703900 27954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:15:28.705825 27954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:15:28.720625 27961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:28.720710 27962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:28.720667 27964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:15:28.721877 27954 server_base.cc:1034] running on GCE node
I20260807 18:15:28.722539 27954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:15:28.723502 27954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:15:28.724753 27954 hybrid_clock.cc:648] HybridClock initialized: now 1786126528724673 us; error 41 us; skew 500 ppm
I20260807 18:15:28.725054 27954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:15:28.727581 27954 webserver.cc:460] Webserver started at http://127.27.4.66:41271/ using document root <none> and password file <none>
I20260807 18:15:28.728314 27954 fs_manager.cc:359] Metadata directory not provided
I20260807 18:15:28.728421 27954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:15:28.728715 27954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:15:28.731531 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/instance:
uuid: "879069a857144936b2103cebd0e62e02"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.732280 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal/instance:
uuid: "879069a857144936b2103cebd0e62e02"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.738215 27954 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 18:15:28.742094 27971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:28.743645 27954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 18:15:28.743865 27954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal
uuid: "879069a857144936b2103cebd0e62e02"
format_stamp: "Formatted at 2026-08-07 18:15:28 on dist-test-slave-bw88"
I20260807 18:15:28.744112 27954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:15:28.777268 27954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:15:28.778324 27954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:15:28.778654 27954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:15:28.779748 27954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:15:28.781430 27954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:15:28.781541 27954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 18:15:28.781699 27954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:15:28.781776 27954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:28.818470 27954 rpc_server.cc:307] RPC server started. Bound to: 127.27.4.66:35099
I20260807 18:15:28.818568 28111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.4.66:35099 every 8 connection(s)
I20260807 18:15:28.819904 27954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb
I20260807 18:15:28.830190 27665 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu as pid 27954
I20260807 18:15:28.830395 27665 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/wal/instance
I20260807 18:15:28.834300 27665 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
/tmp/dist-test-taskOVaSOG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.4.67:0
--local_ip_for_outbound_sockets=127.27.4.67
--webserver_interface=127.27.4.67
--webserver_port=0
--tserver_master_addrs=127.27.4.126:46561
--builtin_ntp_servers=127.27.4.84:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260807 18:15:28.835402 28112 heartbeater.cc:347] Connected to a master server at 127.27.4.126:46561
I20260807 18:15:28.835770 28112 heartbeater.cc:464] Registering TS with master...
I20260807 18:15:28.836642 28112 heartbeater.cc:511] Master 127.27.4.126:46561 requested a full tablet report, sending...
I20260807 18:15:28.838171 27710 ts_manager.cc:194] Registered new tserver with Master: 879069a857144936b2103cebd0e62e02 (127.27.4.66:35099)
I20260807 18:15:28.839033 27710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.4.66:33007
W20260807 18:15:29.182242 28115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 18:15:29.182569 28115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 18:15:29.182739 28115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 18:15:29.190586 28115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 18:15:29.190857 28115 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260807 18:15:29.190997 28115 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260807 18:15:29.191256 28115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.4.67
I20260807 18:15:29.203291 28115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.4.84:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.4.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.4.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.27.4.126:46561
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.4.67
--log_dir=/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 17:13:57 UTC on ccycloud.quasar-mgcilv.root.comops.site
build id 4572599
ASAN enabled
I20260807 18:15:29.205046 28115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 18:15:29.207450 28115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 18:15:29.222558 28124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:29.222514 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 18:15:29.222466 28123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 18:15:29.223444 28115 server_base.cc:1034] running on GCE node
I20260807 18:15:29.224141 28115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 18:15:29.225113 28115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 18:15:29.226364 28115 hybrid_clock.cc:648] HybridClock initialized: now 1786126529226270 us; error 57 us; skew 500 ppm
I20260807 18:15:29.226732 28115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 18:15:29.229202 28115 webserver.cc:460] Webserver started at http://127.27.4.67:43699/ using document root <none> and password file <none>
I20260807 18:15:29.229982 28115 fs_manager.cc:359] Metadata directory not provided
I20260807 18:15:29.230124 28115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 18:15:29.230492 28115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 18:15:29.245229 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/instance:
uuid: "4c5d123b5267475ca291a04ab8c14b9c"
format_stamp: "Formatted at 2026-08-07 18:15:29 on dist-test-slave-bw88"
I20260807 18:15:29.246240 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5d123b5267475ca291a04ab8c14b9c"
format_stamp: "Formatted at 2026-08-07 18:15:29 on dist-test-slave-bw88"
I20260807 18:15:29.252478 28115 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 18:15:29.256605 28132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:29.258152 28115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 18:15:29.258350 28115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal
uuid: "4c5d123b5267475ca291a04ab8c14b9c"
format_stamp: "Formatted at 2026-08-07 18:15:29 on dist-test-slave-bw88"
I20260807 18:15:29.258622 28115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 18:15:29.269681 28115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 18:15:29.270462 28115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 18:15:29.270707 28115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 18:15:29.271611 28115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 18:15:29.273180 28115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 18:15:29.273252 28115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 18:15:29.273381 28115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 18:15:29.273451 28115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 18:15:29.310760 28115 rpc_server.cc:307] RPC server started. Bound to: 127.27.4.67:46537
I20260807 18:15:29.310815 28274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.4.67:46537 every 8 connection(s)
I20260807 18:15:29.312287 28115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/info.pb
I20260807 18:15:29.314127 27665 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOVaSOG/build/asan/bin/kudu as pid 28115
I20260807 18:15:29.314298 27665 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/wal/instance
I20260807 18:15:29.327812 28275 heartbeater.cc:347] Connected to a master server at 127.27.4.126:46561
I20260807 18:15:29.328171 28275 heartbeater.cc:464] Registering TS with master...
I20260807 18:15:29.329058 28275 heartbeater.cc:511] Master 127.27.4.126:46561 requested a full tablet report, sending...
I20260807 18:15:29.330826 27710 ts_manager.cc:194] Registered new tserver with Master: 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.331688 27710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.4.67:35369
I20260807 18:15:29.343693 27665 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 18:15:29.350214 27950 heartbeater.cc:503] Master 127.27.4.126:46561 was elected leader, sending a full tablet report...
I20260807 18:15:29.368291 27710 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52196:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 18:15:29.377017 27710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 18:15:29.414052 28027 tablet_service.cc:1467] Processing CreateTablet for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260807 18:15:29.416118 28027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.417737 28029 tablet_service.cc:1467] Processing CreateTablet for tablet 39eaca12a52347f489cae058c0df12d9 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260807 18:15:29.418617 28029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eaca12a52347f489cae058c0df12d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.414100 28030 tablet_service.cc:1467] Processing CreateTablet for tablet dbe769098d8d420f97f766bab4ec11e0 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260807 18:15:29.418643 28186 tablet_service.cc:1467] Processing CreateTablet for tablet 39eaca12a52347f489cae058c0df12d9 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260807 18:15:29.419740 28030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe769098d8d420f97f766bab4ec11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.419728 28187 tablet_service.cc:1467] Processing CreateTablet for tablet dbe769098d8d420f97f766bab4ec11e0 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260807 18:15:29.418835 28185 tablet_service.cc:1467] Processing CreateTablet for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260807 18:15:29.420634 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe769098d8d420f97f766bab4ec11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.421044 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.421468 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eaca12a52347f489cae058c0df12d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.425741 27862 tablet_service.cc:1467] Processing CreateTablet for tablet 39eaca12a52347f489cae058c0df12d9 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260807 18:15:29.424821 27861 tablet_service.cc:1467] Processing CreateTablet for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260807 18:15:29.424852 27863 tablet_service.cc:1467] Processing CreateTablet for tablet dbe769098d8d420f97f766bab4ec11e0 (DEFAULT_TABLE table=all-types-table [id=a363b1654fb149e282c7a5bf8056eebc]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260807 18:15:29.426937 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eaca12a52347f489cae058c0df12d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.427318 27863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe769098d8d420f97f766bab4ec11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.428541 27861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 18:15:29.445890 28297 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: Bootstrap starting.
I20260807 18:15:29.446090 28301 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Bootstrap starting.
I20260807 18:15:29.448356 28302 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: Bootstrap starting.
I20260807 18:15:29.450829 28301 tablet_bootstrap.cc:654] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.450822 28297 tablet_bootstrap.cc:654] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.452243 28297 log.cc:824] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: Log is configured to *not* fsync() on all Append() calls
I20260807 18:15:29.452245 28301 log.cc:824] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Log is configured to *not* fsync() on all Append() calls
I20260807 18:15:29.452358 28302 tablet_bootstrap.cc:654] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.453815 28302 log.cc:824] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: Log is configured to *not* fsync() on all Append() calls
I20260807 18:15:29.457190 28297 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: No bootstrap required, opened a new log
I20260807 18:15:29.457304 28302 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: No bootstrap required, opened a new log
I20260807 18:15:29.457635 28297 ts_tablet_manager.cc:1397] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260807 18:15:29.457682 28302 ts_tablet_manager.cc:1397] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260807 18:15:29.458303 28301 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: No bootstrap required, opened a new log
I20260807 18:15:29.458580 28301 ts_tablet_manager.cc:1397] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260807 18:15:29.465674 28297 raft_consensus.cc:348] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.466063 28297 raft_consensus.cc:374] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.466141 28297 raft_consensus.cc:729] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879069a857144936b2103cebd0e62e02, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.466747 28297 consensus_queue.cc:260] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.467850 28112 heartbeater.cc:503] Master 127.27.4.126:46561 was elected leader, sending a full tablet report...
I20260807 18:15:29.468650 28297 ts_tablet_manager.cc:1428] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260807 18:15:29.469049 28297 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: Bootstrap starting.
I20260807 18:15:29.469441 28301 raft_consensus.cc:348] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.469448 28302 raft_consensus.cc:348] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.469887 28301 raft_consensus.cc:374] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.469965 28302 raft_consensus.cc:374] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.470032 28301 raft_consensus.cc:729] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5d123b5267475ca291a04ab8c14b9c, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.470189 28302 raft_consensus.cc:729] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70fc93f83824bcca1b7252f86a22dc7, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.470675 28301 consensus_queue.cc:260] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.470917 28297 tablet_bootstrap.cc:654] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.471153 28302 consensus_queue.cc:260] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.477559 28302 ts_tablet_manager.cc:1428] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7: Time spent starting tablet: real 0.020s user 0.016s sys 0.001s
I20260807 18:15:29.479828 28302 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7: Bootstrap starting.
I20260807 18:15:29.482713 28302 tablet_bootstrap.cc:654] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.485438 28297 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: No bootstrap required, opened a new log
I20260807 18:15:29.485639 28297 ts_tablet_manager.cc:1397] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260807 18:15:29.485634 28275 heartbeater.cc:503] Master 127.27.4.126:46561 was elected leader, sending a full tablet report...
I20260807 18:15:29.486415 28301 ts_tablet_manager.cc:1428] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Time spent starting tablet: real 0.028s user 0.017s sys 0.011s
I20260807 18:15:29.486693 28297 raft_consensus.cc:348] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.486949 28297 raft_consensus.cc:374] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.487061 28297 raft_consensus.cc:729] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879069a857144936b2103cebd0e62e02, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.487372 28301 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c: Bootstrap starting.
I20260807 18:15:29.487383 28297 consensus_queue.cc:260] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.487965 28297 ts_tablet_manager.cc:1428] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:15:29.490787 28301 tablet_bootstrap.cc:654] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.492120 28297 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02: Bootstrap starting.
I20260807 18:15:29.494095 28297 tablet_bootstrap.cc:654] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.494266 28301 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c: No bootstrap required, opened a new log
I20260807 18:15:29.494433 28301 ts_tablet_manager.cc:1397] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 18:15:29.494549 28302 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7: No bootstrap required, opened a new log
I20260807 18:15:29.494732 28302 ts_tablet_manager.cc:1397] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260807 18:15:29.495682 28301 raft_consensus.cc:348] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.496024 28301 raft_consensus.cc:374] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.496174 28301 raft_consensus.cc:729] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5d123b5267475ca291a04ab8c14b9c, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.496495 28301 consensus_queue.cc:260] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.496771 28297 tablet_bootstrap.cc:492] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02: No bootstrap required, opened a new log
I20260807 18:15:29.497076 28297 ts_tablet_manager.cc:1397] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 18:15:29.497251 28301 ts_tablet_manager.cc:1428] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260807 18:15:29.497606 28301 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c: Bootstrap starting.
I20260807 18:15:29.497433 28302 raft_consensus.cc:348] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.497748 28302 raft_consensus.cc:374] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.497859 28302 raft_consensus.cc:729] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70fc93f83824bcca1b7252f86a22dc7, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.497876 28297 raft_consensus.cc:348] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.498122 28297 raft_consensus.cc:374] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.498201 28297 raft_consensus.cc:729] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879069a857144936b2103cebd0e62e02, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.498200 28302 consensus_queue.cc:260] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.498440 28297 consensus_queue.cc:260] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.499092 28297 ts_tablet_manager.cc:1428] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 18:15:29.499531 28306 raft_consensus.cc:482] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:15:29.499696 28301 tablet_bootstrap.cc:654] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.499696 28302 ts_tablet_manager.cc:1428] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260807 18:15:29.499835 28306 raft_consensus.cc:504] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.500007 28302 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: Bootstrap starting.
I20260807 18:15:29.501683 28306 leader_election.cc:288] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.502000 28302 tablet_bootstrap.cc:654] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: Neither blocks nor log segments found. Creating new log.
I20260807 18:15:29.503535 28301 tablet_bootstrap.cc:492] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c: No bootstrap required, opened a new log
I20260807 18:15:29.503772 28301 ts_tablet_manager.cc:1397] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260807 18:15:29.504568 28301 raft_consensus.cc:348] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.504906 28301 raft_consensus.cc:374] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.505023 28301 raft_consensus.cc:729] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5d123b5267475ca291a04ab8c14b9c, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.505306 28301 consensus_queue.cc:260] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.505631 28302 tablet_bootstrap.cc:492] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: No bootstrap required, opened a new log
I20260807 18:15:29.505822 28302 ts_tablet_manager.cc:1397] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 18:15:29.506687 28301 ts_tablet_manager.cc:1428] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 18:15:29.507826 28302 raft_consensus.cc:348] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.508142 28302 raft_consensus.cc:374] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 18:15:29.508251 28302 raft_consensus.cc:729] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70fc93f83824bcca1b7252f86a22dc7, State: Initialized, Role: FOLLOWER
I20260807 18:15:29.508575 28302 consensus_queue.cc:260] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.509291 28302 ts_tablet_manager.cc:1428] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 18:15:29.520510 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eaca12a52347f489cae058c0df12d9" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c" is_pre_election: true
I20260807 18:15:29.521214 28214 raft_consensus.cc:2454] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 0.
I20260807 18:15:29.521237 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eaca12a52347f489cae058c0df12d9" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70fc93f83824bcca1b7252f86a22dc7" is_pre_election: true
I20260807 18:15:29.521886 27888 raft_consensus.cc:2454] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 0.
I20260807 18:15:29.522008 27975 leader_election.cc:302] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5d123b5267475ca291a04ab8c14b9c, 879069a857144936b2103cebd0e62e02; no voters:
I20260807 18:15:29.522465 28306 raft_consensus.cc:2781] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:15:29.522629 28306 raft_consensus.cc:482] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:15:29.522746 28306 raft_consensus.cc:3037] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.524883 28306 raft_consensus.cc:504] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.525556 28306 leader_election.cc:288] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 election: Requested vote from peers b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.525986 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eaca12a52347f489cae058c0df12d9" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70fc93f83824bcca1b7252f86a22dc7"
I20260807 18:15:29.526047 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eaca12a52347f489cae058c0df12d9" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c"
I20260807 18:15:29.526160 27888 raft_consensus.cc:3037] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.526325 28214 raft_consensus.cc:3037] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.528368 27888 raft_consensus.cc:2454] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 1.
I20260807 18:15:29.528625 28214 raft_consensus.cc:2454] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 1.
I20260807 18:15:29.528836 27975 leader_election.cc:302] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879069a857144936b2103cebd0e62e02, b70fc93f83824bcca1b7252f86a22dc7; no voters:
I20260807 18:15:29.529138 28306 raft_consensus.cc:2781] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:15:29.529651 28306 raft_consensus.cc:686] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [term 1 LEADER]: Becoming Leader. State: Replica: 879069a857144936b2103cebd0e62e02, State: Running, Role: LEADER
I20260807 18:15:29.530149 28306 consensus_queue.cc:237] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.535156 27710 catalog_manager.cc:5495] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 879069a857144936b2103cebd0e62e02 (127.27.4.66). New cstate: current_term: 1 leader_uuid: "879069a857144936b2103cebd0e62e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } health_report { overall_health: UNKNOWN } } }
I20260807 18:15:29.546367 28306 raft_consensus.cc:482] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:15:29.546649 28306 raft_consensus.cc:504] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.547425 28306 leader_election.cc:288] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.548059 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a1d8dfdd9649978f2c7d8f5ed4d72a" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70fc93f83824bcca1b7252f86a22dc7" is_pre_election: true
I20260807 18:15:29.548370 27888 raft_consensus.cc:2454] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 0.
I20260807 18:15:29.548236 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a1d8dfdd9649978f2c7d8f5ed4d72a" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c" is_pre_election: true
I20260807 18:15:29.548596 28214 raft_consensus.cc:2454] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 0.
I20260807 18:15:29.548834 27975 leader_election.cc:302] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879069a857144936b2103cebd0e62e02, b70fc93f83824bcca1b7252f86a22dc7; no voters:
I20260807 18:15:29.549198 28306 raft_consensus.cc:2781] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:15:29.549310 28306 raft_consensus.cc:482] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:15:29.549482 28306 raft_consensus.cc:3037] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.551416 28306 raft_consensus.cc:504] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.551970 28306 leader_election.cc:288] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 election: Requested vote from peers b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.552428 27888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a1d8dfdd9649978f2c7d8f5ed4d72a" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70fc93f83824bcca1b7252f86a22dc7"
I20260807 18:15:29.552549 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a1d8dfdd9649978f2c7d8f5ed4d72a" candidate_uuid: "879069a857144936b2103cebd0e62e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c"
I20260807 18:15:29.552675 27888 raft_consensus.cc:3037] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.552834 28214 raft_consensus.cc:3037] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.554716 27888 raft_consensus.cc:2454] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P b70fc93f83824bcca1b7252f86a22dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 1.
I20260807 18:15:29.554867 28214 raft_consensus.cc:2454] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 4c5d123b5267475ca291a04ab8c14b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879069a857144936b2103cebd0e62e02 in term 1.
I20260807 18:15:29.555228 27975 leader_election.cc:302] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879069a857144936b2103cebd0e62e02, b70fc93f83824bcca1b7252f86a22dc7; no voters:
I20260807 18:15:29.555529 28306 raft_consensus.cc:2781] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:15:29.555727 28306 raft_consensus.cc:686] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [term 1 LEADER]: Becoming Leader. State: Replica: 879069a857144936b2103cebd0e62e02, State: Running, Role: LEADER
I20260807 18:15:29.556026 28306 consensus_queue.cc:237] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } }
I20260807 18:15:29.558768 27710 catalog_manager.cc:5495] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 879069a857144936b2103cebd0e62e02 (127.27.4.66). New cstate: current_term: 1 leader_uuid: "879069a857144936b2103cebd0e62e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } health_report { overall_health: UNKNOWN } } }
W20260807 18:15:29.564314 28276 tablet.cc:2369] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:15:29.572177 28113 tablet.cc:2369] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 18:15:29.578011 27951 tablet.cc:2369] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 18:15:29.597967 28308 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 18:15:29.598228 28308 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.600031 28308 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 879069a857144936b2103cebd0e62e02 (127.27.4.66:35099), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.608781 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe769098d8d420f97f766bab4ec11e0" candidate_uuid: "b70fc93f83824bcca1b7252f86a22dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c" is_pre_election: true
I20260807 18:15:29.609099 28214 raft_consensus.cc:2454] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70fc93f83824bcca1b7252f86a22dc7 in term 0.
I20260807 18:15:29.609061 28053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe769098d8d420f97f766bab4ec11e0" candidate_uuid: "b70fc93f83824bcca1b7252f86a22dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879069a857144936b2103cebd0e62e02" is_pre_election: true
I20260807 18:15:29.609661 28053 raft_consensus.cc:2454] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70fc93f83824bcca1b7252f86a22dc7 in term 0.
I20260807 18:15:29.609731 27806 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7; no voters:
I20260807 18:15:29.610303 28308 raft_consensus.cc:2781] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 18:15:29.610495 28308 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 18:15:29.610627 28308 raft_consensus.cc:3037] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.612977 28308 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.613828 28308 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [CANDIDATE]: Term 1 election: Requested vote from peers 879069a857144936b2103cebd0e62e02 (127.27.4.66:35099), 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537)
I20260807 18:15:29.614280 28053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe769098d8d420f97f766bab4ec11e0" candidate_uuid: "b70fc93f83824bcca1b7252f86a22dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879069a857144936b2103cebd0e62e02"
I20260807 18:15:29.614408 28214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe769098d8d420f97f766bab4ec11e0" candidate_uuid: "b70fc93f83824bcca1b7252f86a22dc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5d123b5267475ca291a04ab8c14b9c"
I20260807 18:15:29.614511 28053 raft_consensus.cc:3037] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.614674 28214 raft_consensus.cc:3037] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 18:15:29.616703 28214 raft_consensus.cc:2454] T dbe769098d8d420f97f766bab4ec11e0 P 4c5d123b5267475ca291a04ab8c14b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70fc93f83824bcca1b7252f86a22dc7 in term 1.
I20260807 18:15:29.616947 28053 raft_consensus.cc:2454] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70fc93f83824bcca1b7252f86a22dc7 in term 1.
I20260807 18:15:29.617146 27806 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7; no voters:
I20260807 18:15:29.617494 28308 raft_consensus.cc:2781] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 18:15:29.617998 28308 raft_consensus.cc:686] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [term 1 LEADER]: Becoming Leader. State: Replica: b70fc93f83824bcca1b7252f86a22dc7, State: Running, Role: LEADER
I20260807 18:15:29.618523 28308 consensus_queue.cc:237] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:29.623227 27710 catalog_manager.cc:5495] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65). New cstate: current_term: 1 leader_uuid: "b70fc93f83824bcca1b7252f86a22dc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260807 18:15:29.913694 28306 consensus_queue.cc:1035] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:29.924374 28306 consensus_queue.cc:1035] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:30.042176 28330 consensus_queue.cc:1035] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:30.048818 28330 consensus_queue.cc:1035] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:30.107725 28308 consensus_queue.cc:1035] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:30.114251 28308 consensus_queue.cc:1035] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 18:15:31.078394 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling FlushMRSOp(39eaca12a52347f489cae058c0df12d9): perf score=1.000000
W20260807 18:15:31.231299 28281 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.4.66:35099, user_credentials={real_user=slave}} blocked reactor thread for 102173us
I20260807 18:15:31.588634 27951 maintenance_manager.cc:419] P b70fc93f83824bcca1b7252f86a22dc7: Scheduling FlushMRSOp(dbe769098d8d420f97f766bab4ec11e0): perf score=1.000000
I20260807 18:15:31.825901 28276 maintenance_manager.cc:419] P 4c5d123b5267475ca291a04ab8c14b9c: Scheduling FlushMRSOp(39eaca12a52347f489cae058c0df12d9): perf score=1.000000
W20260807 18:15:31.981678 28279 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.4.65:35371, user_credentials={real_user=slave}} blocked reactor thread for 231151us
W20260807 18:15:32.113396 27811 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/data (489349120 bytes available vs 518351011 bytes reserved) (error 28)
W20260807 18:15:32.120191 28137 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-2/data/data (455790592 bytes available vs 518351011 bytes reserved) (error 28)
W20260807 18:15:32.152221 27811 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-0/data/data (455802880 bytes available vs 518351011 bytes reserved) (error 28)
W20260807 18:15:32.176994 28137 tablet_replica_mm_ops.cc:214] T 39eaca12a52347f489cae058c0df12d9 P 4c5d123b5267475ca291a04ab8c14b9c: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 39eaca12a52347f489cae058c0df12d9: unable to open output file for column double_val DOUBLE NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 39eaca12a52347f489cae058c0df12d9's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260807 18:15:32.177302 28137 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 39eaca12a52347f489cae058c0df12d9: unable to open output file for column double_val DOUBLE NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 39eaca12a52347f489cae058c0df12d9's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f99e8ef19ad google::LogMessage::Fail() at ??:0
@ 0x7f99e8ef488a google::LogMessage::SendToLog() at ??:0
@ 0x7f99e8ef14de google::LogMessage::Flush() at ??:0
@ 0x7f99e8ef2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f99f82c6e7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f99eb9df090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f99eb9e1e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f99eb9e1ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f99ebb3d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f99ebb4091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f99ebb40799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f99ebb1beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f99eedf56db start_thread at ??:0
@ 0x7f99e7d8b61f clone at ??:0
W20260807 18:15:32.777740 27806 connection.cc:535] client connection to 127.27.4.67:46537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 18:15:32.778103 27806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 18:15:32.786201 27975 connection.cc:535] client connection to 127.27.4.67:46537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 18:15:32.786535 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 18:15:32.815255 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 18:15:32.815630 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 18:15:32.833495 27806 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.4.67:46537, user_credentials={real_user=slave}} blocked reactor thread for 55414.9us
W20260807 18:15:32.842703 27806 consensus_peers.cc:487] T dbe769098d8d420f97f766bab4ec11e0 P b70fc93f83824bcca1b7252f86a22dc7 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 18:15:33.016108 28281 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.4.66:35099, user_credentials={real_user=slave}} blocked reactor thread for 55796.9us
I20260807 18:15:33.531131 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: FlushMRSOp(39eaca12a52347f489cae058c0df12d9) complete. Timing: real 2.451s user 0.703s sys 0.173s Metrics: {"bytes_written":221748,"cfile_init":1,"compiler_manager_pool.queue_time_us":29263,"dirs.queue_time_us":38009,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4192,"lbm_writes_lt_1ms":81,"peak_mem_usage":0,"rows_written":4768,"spinlock_wait_cycles":253184,"thread_start_us":74986,"threads_started":2}
I20260807 18:15:33.533246 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling UndoDeltaBlockGCOp(39eaca12a52347f489cae058c0df12d9): 30089 bytes on disk
I20260807 18:15:33.536113 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: UndoDeltaBlockGCOp(39eaca12a52347f489cae058c0df12d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20260807 18:15:33.537036 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling FlushMRSOp(dbe769098d8d420f97f766bab4ec11e0): perf score=1.937278
I20260807 18:15:33.529240 28406 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41318 (ReqId={client: d653197743584325a366d180933f5248, seq_no=3, attempt_no=0}) took 1098 ms. Trace:
I20260807 18:15:33.541574 28406 rpcz_store.cc:276] 0807 18:15:32.430676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 18:15:32.430795 (+ 119us) service_pool.cc:227] Handling call
0807 18:15:33.529181 (+1098386us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0807 18:15:32.432817 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 39eaca12a52347f489cae058c0df12d9
0807 18:15:32.433052 (+ 235us) write_op.cc:432] Acquiring schema lock in shared mode
0807 18:15:32.433060 (+ 8us) write_op.cc:435] Acquired schema lock
0807 18:15:32.433063 (+ 3us) tablet.cc:660] Decoding operations
0807 18:15:32.846329 (+413266us) write_op.cc:620] Acquiring the partition lock for write op
0807 18:15:32.846349 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0807 18:15:32.846357 (+ 8us) tablet.cc:683] Acquiring locks for 5232 operations
0807 18:15:32.904527 (+ 58170us) tablet.cc:699] Row locks acquired
0807 18:15:32.904532 (+ 5us) write_op.cc:260] PREPARE: finished
0807 18:15:32.904636 (+ 104us) write_op.cc:270] Start()
0807 18:15:32.904696 (+ 60us) write_op.cc:276] Timestamp: P: 1786126532904598 usec, L: 0
0807 18:15:32.904700 (+ 4us) op_driver.cc:331] REPLICATION: starting
0807 18:15:32.906169 (+ 1469us) log.cc:842] Serialized 827294 byte log entry
0807 18:15:32.983519 (+ 77350us) op_driver.cc:444] REPLICATION: finished
0807 18:15:32.983935 (+ 416us) write_op.cc:301] APPLY: starting
0807 18:15:32.983959 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0807 18:15:33.011226 (+ 27267us) tablet.cc:1368] finished BulkCheckPresence
0807 18:15:33.011279 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0807 18:15:33.425965 (+414686us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x61700008038c. stack: 00007fce2451f66e 00007fce2451f430 00007fce235a65ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00000000009652d0 0000000000a6825c 00007fce30be375c 00007fce30be2daf 00007fce30d8e27f 00007fce30d6448f 00007fce30d6777b 00007fce30d675f8 00000000009edb7c 00007fce24565344
0807 18:15:33.501510 (+ 75545us) tablet.cc:1381] finished ApplyRowOperation cycle
0807 18:15:33.504672 (+ 3162us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0807 18:15:33.504681 (+ 9us) write_op.cc:312] APPLY: finished
0807 18:15:33.525159 (+ 20478us) log.cc:842] Serialized 31414 byte log entry
0807 18:15:33.525325 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0807 18:15:33.528751 (+ 3426us) write_op.cc:454] Released schema lock
0807 18:15:33.529032 (+ 281us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":320,"num_ops":5232,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":146983,"prepare.run_wall_time_us":520179,"raft.queue_time_us":153,"raft.run_cpu_time_us":1733,"raft.run_wall_time_us":1734,"replication_time_us":78782,"spinlock_wait_cycles":23340800,"thread_start_us":46517,"threads_started":2,"wal-append.queue_time_us":13418}]]}
W20260807 18:15:33.712199 28281 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.4.66:35099, user_credentials={real_user=slave}} blocked reactor thread for 168642us
I20260807 18:15:34.280707 27811 maintenance_manager.cc:643] P b70fc93f83824bcca1b7252f86a22dc7: FlushMRSOp(dbe769098d8d420f97f766bab4ec11e0) complete. Timing: real 2.691s user 1.286s sys 0.268s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":22281,"compiler_manager_pool.run_cpu_time_us":1045601,"compiler_manager_pool.run_wall_time_us":2168872,"dirs.queue_time_us":478,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":878,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5326,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":34492,"threads_started":3,"wal-append.queue_time_us":12224}
I20260807 18:15:34.282982 27951 maintenance_manager.cc:419] P b70fc93f83824bcca1b7252f86a22dc7: Scheduling FlushMRSOp(39eaca12a52347f489cae058c0df12d9): perf score=1.937278
W20260807 18:15:34.284793 27811 tablet_replica_mm_ops.cc:214] T 39eaca12a52347f489cae058c0df12d9 P b70fc93f83824bcca1b7252f86a22dc7: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 39eaca12a52347f489cae058c0df12d9: unable to open output file for column key INT32 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 39eaca12a52347f489cae058c0df12d9's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260807 18:15:34.285043 27811 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 39eaca12a52347f489cae058c0df12d9: unable to open output file for column key INT32 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 39eaca12a52347f489cae058c0df12d9's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7fe24b7459ad google::LogMessage::Fail() at ??:0
@ 0x7fe24b74888a google::LogMessage::SendToLog() at ??:0
@ 0x7fe24b7454de google::LogMessage::Flush() at ??:0
@ 0x7fe24b746a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe25ab1ae7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7fe24e233090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7fe24e235e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7fe24e235ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe24e391345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe24e39491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe24e394799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe24e36feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe2516496db start_thread at ??:0
@ 0x7fe24a5df61f clone at ??:0
W20260807 18:15:34.421092 28281 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.27.4.66:35099, user_credentials={real_user=slave}} blocked reactor thread for 69915.4us
I20260807 18:15:34.541853 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:15:34.620520 27975 connection.cc:535] client connection to 127.27.4.65:35371 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 18:15:34.622112 28494 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 18:15:34.526768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.4.65:35371 (local address 127.0.0.1:45796)
0807 18:15:34.529503 (+ 2735us) negotiation.cc:107] Waiting for socket to connect
0807 18:15:34.529518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 18:15:34.529633 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 18:15:34.620223 (+ 90590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2607,"thread_start_us":589,"threads_started":1}
W20260807 18:15:34.622095 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:34.634819 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:34.635682 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 18:15:34.845170 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260807 18:15:34.882187 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: FlushMRSOp(dbe769098d8d420f97f766bab4ec11e0) complete. Timing: real 1.345s user 1.072s sys 0.126s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":229563,"compiler_manager_pool.run_cpu_time_us":75,"compiler_manager_pool.run_wall_time_us":76,"dirs.queue_time_us":476,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":1649,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5504,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":85,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":1884,"threads_started":2,"wal-append.queue_time_us":1727}
I20260807 18:15:34.883494 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling UndoDeltaBlockGCOp(dbe769098d8d420f97f766bab4ec11e0): 62919 bytes on disk
I20260807 18:15:34.884182 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: UndoDeltaBlockGCOp(dbe769098d8d420f97f766bab4ec11e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20260807 18:15:34.884967 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling FlushMRSOp(b7a1d8dfdd9649978f2c7d8f5ed4d72a): perf score=1.937278
I20260807 18:15:34.890261 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:34.936044 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:35.010860 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260807 18:15:35.268074 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:35.313239 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:15:35.578567 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: FlushMRSOp(b7a1d8dfdd9649978f2c7d8f5ed4d72a) complete. Timing: real 0.693s user 0.612s sys 0.079s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":417,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1416,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3576,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":495,"threads_started":2,"wal-append.queue_time_us":334}
I20260807 18:15:35.579344 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling UndoDeltaBlockGCOp(b7a1d8dfdd9649978f2c7d8f5ed4d72a): 63930 bytes on disk
I20260807 18:15:35.579895 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: UndoDeltaBlockGCOp(b7a1d8dfdd9649978f2c7d8f5ed4d72a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20260807 18:15:35.580477 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling FlushMRSOp(39eaca12a52347f489cae058c0df12d9): perf score=1.000000
I20260807 18:15:35.741392 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:35.776103 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:15:35.778723 28528 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:35.778924 28528 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:35.779512 28528 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:35.781330 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:35.781590 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:35.781719 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:35.781970 28528 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:15:35.938444 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: FlushMRSOp(39eaca12a52347f489cae058c0df12d9) complete. Timing: real 0.358s user 0.313s sys 0.037s Metrics: {"bytes_written":246650,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2354,"lbm_writes_lt_1ms":82,"peak_mem_usage":0,"rows_written":5232,"thread_start_us":327,"threads_started":1,"wal-append.queue_time_us":466}
I20260807 18:15:35.939813 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling CompactRowSetsOp(39eaca12a52347f489cae058c0df12d9): perf score=0.014857
I20260807 18:15:36.355062 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: CompactRowSetsOp(39eaca12a52347f489cae058c0df12d9) complete. Timing: real 0.415s user 0.396s sys 0.017s Metrics: {"bytes_written":472622,"cfile_cache_hit":40,"cfile_cache_hit_bytes":398360,"cfile_cache_miss":116,"cfile_cache_miss_bytes":602482,"cfile_init":32,"delta_iterators_relevant":4,"dirs.queue_time_us":9815,"dirs.run_cpu_time_us":4033,"dirs.run_wall_time_us":21881,"drs_written":1,"lbm_read_time_us":3390,"lbm_reads_lt_1ms":244,"lbm_write_time_us":3246,"lbm_writes_lt_1ms":86,"mutex_wait_us":473,"num_input_rowsets":2,"peak_mem_usage":102942,"rows_written":10000,"spinlock_wait_cycles":14720,"thread_start_us":7478,"threads_started":7}
I20260807 18:15:36.356844 28113 maintenance_manager.cc:419] P 879069a857144936b2103cebd0e62e02: Scheduling UndoDeltaBlockGCOp(39eaca12a52347f489cae058c0df12d9): 63428 bytes on disk
I20260807 18:15:36.357511 27980 maintenance_manager.cc:643] P 879069a857144936b2103cebd0e62e02: UndoDeltaBlockGCOp(39eaca12a52347f489cae058c0df12d9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20260807 18:15:36.373278 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:36.405364 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:36.739332 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 18:15:37.243770 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260807 18:15:37.251411 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:37.283135 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:37.314937 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 18:15:37.751663 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260807 18:15:37.776422 28528 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:37.776633 28528 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:37.777311 28528 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:37.779320 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:37.779515 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:37.779611 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:37.779819 28528 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:15:37.827385 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260807 18:15:38.367544 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:38.398252 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:39.026814 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 18:15:39.414911 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260807 18:15:39.490211 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:39.521104 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:15:39.732391 28528 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:39.732637 28528 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:39.733405 28528 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:39.735371 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:39.735584 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:39.735685 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:39.735934 28528 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:15:39.883908 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260807 18:15:40.434935 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260807 18:15:40.608724 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:40.642088 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:15:41.732358 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:15:41.748529 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:41.764417 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:41.878412 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260807 18:15:42.427870 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260807 18:15:42.845899 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:42.878358 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:42.890265 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:15:43.034643 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260807 18:15:43.307489 28776 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:43.307825 28776 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:43.308496 28776 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:43.310721 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:43.310992 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:43.311113 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:43.311378 28776 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:15:43.960830 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:43.992419 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:44.148552 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260807 18:15:44.376757 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260807 18:15:44.997886 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260807 18:15:45.083623 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:45.118577 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:45.857066 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260807 18:15:46.203091 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:46.234092 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:46.553169 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260807 18:15:47.038167 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260807 18:15:47.319257 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:47.354919 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:47.419997 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260807 18:15:47.912894 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20260807 18:15:48.242486 28776 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:48.242700 28776 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:48.243261 28776 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:48.245301 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:48.245481 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:48.245581 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:48.245934 28776 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:15:48.323760 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260807 18:15:48.441865 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:48.474654 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:48.905961 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260807 18:15:49.461998 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260807 18:15:49.560519 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:49.595005 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:49.690330 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260807 18:15:50.615881 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260807 18:15:50.691587 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260807 18:15:50.699127 28776 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:15:50.699308 28776 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:15:50.699932 28776 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:15:50.702020 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:15:50.702212 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:15:50.702323 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:15:50.702692 28776 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:50.724350 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:51.399026 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260807 18:15:51.815681 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:51.845968 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:51.904148 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260807 18:15:52.219667 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260807 18:15:52.928830 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:52.960839 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:53.019827 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260807 18:15:53.308296 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260807 18:15:53.710048 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260807 18:15:54.045109 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:54.077565 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:54.649294 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260807 18:15:54.792991 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260807 18:15:55.169973 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:55.204097 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:55.810420 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260807 18:15:55.898801 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260807 18:15:56.301020 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:56.333849 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:56.972150 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260807 18:15:57.226167 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260807 18:15:57.427852 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:57.463914 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:58.217399 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:15:58.218462 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260807 18:15:58.431236 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260807 18:15:58.553298 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:58.585800 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:59.444180 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260807 18:15:59.672477 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:15:59.706249 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:15:59.907864 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260807 18:16:00.724467 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260807 18:16:00.795543 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:00.828328 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:01.069566 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260807 18:16:01.819267 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260807 18:16:01.925534 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:01.957283 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:02.460793 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260807 18:16:03.044850 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:03.077220 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:03.247707 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:16:03.248755 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260807 18:16:03.563098 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260807 18:16:04.171514 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:04.212458 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:04.513303 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260807 18:16:05.058382 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260807 18:16:05.300280 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:05.331405 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:05.666968 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260807 18:16:06.355192 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260807 18:16:06.420398 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:06.453290 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:06.856441 28776 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:16:06.856709 28776 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:16:06.857443 28776 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:16:06.859442 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:16:06.859654 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:16:06.859769 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:16:06.859993 28776 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:16:06.964381 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260807 18:16:07.538327 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260807 18:16:07.555074 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:07.589354 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:08.288211 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:16:08.289443 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260807 18:16:08.682682 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:08.715054 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:08.896029 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260807 18:16:09.693990 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260807 18:16:09.805784 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:09.838945 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:10.090346 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260807 18:16:10.925958 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:16:10.928660 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:10.956604 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:11.482816 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260807 18:16:12.041190 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:12.071784 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:12.252772 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260807 18:16:12.556733 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260807 18:16:13.164469 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:13.196825 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:13.377035 28776 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:16:13.377271 28776 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:16:13.378090 28776 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:16:13.379413 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:16:13.380594 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:16:13.380832 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:16:13.380939 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:16:13.381207 28776 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:16:13.399039 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260807 18:16:13.636003 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260807 18:16:14.282595 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:14.313681 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:14.554622 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260807 18:16:15.042645 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260807 18:16:15.402873 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:15.434708 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:15.643131 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260807 18:16:16.041323 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260807 18:16:16.517112 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:16.557037 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:17.085660 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260807 18:16:17.569429 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260807 18:16:17.645270 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:17.676509 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:18.101720 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260807 18:16:18.430078 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:16:18.454651 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260807 18:16:18.767136 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:18.800230 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:19.380252 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260807 18:16:19.887107 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:19.919978 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:20.206283 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260807 18:16:20.637104 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260807 18:16:21.011189 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:16:21.014631 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:21.044194 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:21.689157 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260807 18:16:22.128911 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:22.160246 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:22.416046 29876 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:16:22.416240 29876 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:16:22.416822 29876 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:16:22.418749 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:16:22.418951 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:16:22.419046 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:16:22.419313 29876 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:16:22.572926 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260807 18:16:23.215911 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260807 18:16:23.250501 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:23.282919 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:23.315128 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260807 18:16:23.433727 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:16:24.282815 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260807 18:16:24.379812 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:24.411819 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:24.853955 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260807 18:16:25.500221 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:25.532640 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:25.952221 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260807 18:16:25.952868 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260807 18:16:26.617587 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:26.651618 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:26.749307 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260807 18:16:27.462656 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260807 18:16:27.743940 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:27.772888 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:28.541457 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:16:28.542467 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260807 18:16:28.721421 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260807 18:16:28.863332 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:28.894722 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:28.981895 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260807 18:16:29.980373 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:30.020463 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:30.144724 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260807 18:16:31.109122 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260807 18:16:31.113682 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:31.146032 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:31.193497 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260807 18:16:31.468860 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260807 18:16:32.229822 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:32.259933 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:32.585067 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260807 18:16:33.343437 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:33.375279 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:33.604794 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:16:33.605916 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260807 18:16:33.748872 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260807 18:16:33.792959 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260807 18:16:34.464146 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:34.495769 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:35.447896 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260807 18:16:35.590857 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:35.622421 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:36.020826 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260807 18:16:36.401968 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260807 18:16:36.403705 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260807 18:16:36.706180 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
I20260807 18:16:36.735347 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
I20260807 18:16:36.735587 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:36.736325 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:16:36.738546 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:16:36.738745 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:16:36.738847 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:16:36.739257 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:16:36.739837 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:37.824268 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:37.856027 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:38.086875 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260807 18:16:38.648128 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:16:38.649101 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260807 18:16:38.823621 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260807 18:16:38.906255 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260807 18:16:38.940753 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:38.972405 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:40.056139 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:40.087512 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:40.668502 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260807 18:16:41.177524 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:41.207861 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:41.268661 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260807 18:16:41.359221 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260807 18:16:41.551311 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260807 18:16:42.298382 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:42.332685 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:43.419103 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
W20260807 18:16:43.446604 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:43.450424 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:43.771519 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:16:43.772445 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260807 18:16:44.010987 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260807 18:16:44.094769 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260807 18:16:44.533783 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:44.566056 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:45.652710 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:45.684508 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:45.732389 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260807 18:16:46.228521 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260807 18:16:46.293813 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260807 18:16:46.664223 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260807 18:16:46.780753 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:46.822993 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:47.905074 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:47.936561 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:48.196009 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260807 18:16:48.687795 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260807 18:16:48.821223 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:16:48.822304 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260807 18:16:49.026152 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:49.058570 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:49.142084 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260807 18:16:49.766619 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:16:49.766835 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:16:49.767427 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:16:49.768810 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:16:49.769018 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:16:49.769138 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:16:49.769419 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:16:50.148921 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:50.178351 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:50.682878 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260807 18:16:51.076962 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260807 18:16:51.262014 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:51.292483 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:51.373890 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260807 18:16:51.621976 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260807 18:16:52.381029 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:52.413292 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:53.286216 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260807 18:16:53.500522 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:53.533838 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:53.662331 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260807 18:16:53.767584 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260807 18:16:53.852743 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:16:53.853863 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260807 18:16:54.622080 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:54.653095 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:55.738565 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:55.773008 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:56.046968 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260807 18:16:56.185086 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260807 18:16:56.217506 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260807 18:16:56.296407 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260807 18:16:56.861555 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:56.893015 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:16:57.982578 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:58.014914 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:16:58.476099 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260807 18:16:58.504956 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260807 18:16:58.548049 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260807 18:16:58.826160 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260807 18:16:58.897143 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260807 18:16:59.108357 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:16:59.140964 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:00.230332 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:00.264230 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:00.648025 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260807 18:17:00.874184 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260807 18:17:00.990221 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260807 18:17:01.350798 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:01.381657 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:01.483840 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260807 18:17:02.467950 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:02.501678 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:03.185142 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260807 18:17:03.394634 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260807 18:17:03.589869 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:03.632160 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:03.694828 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260807 18:17:03.888054 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260807 18:17:03.919025 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260807 18:17:04.720721 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:04.753767 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:05.842653 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:17:05.852586 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:05.881600 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:05.883386 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260807 18:17:06.056433 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:06.056664 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:06.057240 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:06.059197 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:06.059383 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:06.059466 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:06.059752 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:17:06.108551 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260807 18:17:06.559798 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260807 18:17:06.968330 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:07.001744 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:08.086668 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:08.119218 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:08.345654 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260807 18:17:08.475903 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260807 18:17:08.767278 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260807 18:17:08.993920 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:17:08.994850 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260807 18:17:09.213562 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:09.244334 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:09.349469 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:09.349728 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:09.350574 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:09.352658 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:09.352859 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:09.352953 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:09.353200 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 18:17:10.327375 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:10.361245 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:10.887331 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260807 18:17:11.011793 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260807 18:17:11.151476 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260807 18:17:11.162809 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260807 18:17:11.444877 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:11.475347 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:12.558317 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:12.589687 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:13.326156 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260807 18:17:13.617849 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260807 18:17:13.671765 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:13.703990 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:13.754961 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260807 18:17:13.777143 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260807 18:17:14.138662 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260807 18:17:14.797304 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:14.830775 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:15.535355 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260807 18:17:15.936013 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260807 18:17:15.958366 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:15.970043 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:16.016744 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260807 18:17:16.251152 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260807 18:17:17.055375 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:17.088536 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:18.061292 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260807 18:17:18.171938 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:18.202889 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:18.331322 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260807 18:17:18.457381 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260807 18:17:18.555143 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260807 18:17:19.290078 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:19.321682 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:19.394675 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20260807 18:17:20.415961 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:20.445547 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:20.499310 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260807 18:17:20.674783 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260807 18:17:20.977533 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260807 18:17:21.046653 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260807 18:17:21.532096 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:21.565369 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:22.658407 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:22.689558 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:23.007813 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260807 18:17:23.169440 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260807 18:17:23.586220 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260807 18:17:23.616217 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260807 18:17:23.776842 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:23.809007 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:24.399627 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260807 18:17:24.895417 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:24.930133 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260807 18:17:25.215441 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:25.215700 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:25.216403 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:25.218453 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:25.218657 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:25.218734 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:25.218940 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:17:25.323978 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260807 18:17:25.862546 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260807 18:17:25.956557 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260807 18:17:26.026343 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:26.059719 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:26.214880 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260807 18:17:27.152601 28027 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
767d100e67004aa29f1b558de36175c0 | 127.27.4.126:46561 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.27.4.84:45537 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
879069a857144936b2103cebd0e62e02 | 127.27.4.66:35099 | HEALTHY | <none> | 2 | 0
4c5d123b5267475ca291a04ab8c14b9c | 127.27.4.67:46537 | UNAVAILABLE | <none> | n/a | n/a
b70fc93f83824bcca1b7252f86a22dc7 | 127.27.4.65:35371 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.27.4.67:46537: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.27.4.65:35371: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.27.4.66:35099
ipki_server_key_size | 768 | experimental | 127.27.4.66:35099
local_ip_for_outbound_sockets | 127.27.4.66 | experimental | 127.27.4.66:35099
never_fsync | true | unsafe,advanced | 127.27.4.66:35099
openssl_security_level_override | 0 | unsafe,hidden | 127.27.4.66:35099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.27.4.66:35099
server_dump_info_format | pb | hidden | 127.27.4.66:35099
server_dump_info_path | /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/ts-1/data/info.pb | hidden | 127.27.4.66:35099
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.27.4.66:35099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------
builtin_ntp_servers | 127.27.4.84:45537 | 127.27.4.66:35099
time_source | builtin | 127.27.4.66:35099
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.27.4.126:46561, tserver@127.27.4.66:35099
Tablet Summary
Tablet dbe769098d8d420f97f766bab4ec11e0 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable [LEADER]
All reported replicas are:
A = 879069a857144936b2103cebd0e62e02
B = 4c5d123b5267475ca291a04ab8c14b9c
C = b70fc93f83824bcca1b7252f86a22dc7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 39eaca12a52347f489cae058c0df12d9 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet b7a1d8dfdd9649978f2c7d8f5ed4d72a of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): TS unavailable
879069a857144936b2103cebd0e62e02 (127.27.4.66:35099): RUNNING [LEADER]
4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): TS unavailable
All reported replicas are:
A = b70fc93f83824bcca1b7252f86a22dc7
B = 879069a857144936b2103cebd0e62e02
C = 4c5d123b5267475ca291a04ab8c14b9c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260807 18:17:27.185777 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260807 18:17:27.862460 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260807 18:17:27.918701 27697 maintenance_manager.cc:419] P 767d100e67004aa29f1b558de36175c0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033371
W20260807 18:17:27.922034 27696 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskOVaSOG/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786126527143970-27665-0/minicluster-data/master-0/data/data (453304320 bytes available vs 518351011 bytes reserved) (error 28)
W20260807 18:17:27.922897 27696 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 767d100e67004aa29f1b558de36175c0: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260807 18:17:27.923023 27696 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f979384b9ad google::LogMessage::Fail() at ??:0
@ 0x7f979384e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f979384b4de google::LogMessage::Flush() at ??:0
@ 0x7f979384ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f97a2c20e7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f9796339090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f979633be53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f979633bce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9796497345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f979649a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f979649a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9796475eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f979974f6db start_thread at ??:0
@ 0x7f97926e561f clone at ??:0
W20260807 18:17:28.417977 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260807 18:17:28.481910 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260807 18:17:28.484239 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260807 18:17:28.526871 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:17:29.408172 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:17:30.329106 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260807 18:17:30.917379 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260807 18:17:31.032048 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260807 18:17:31.107963 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260807 18:17:32.608750 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260807 18:17:33.288383 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260807 18:17:33.623422 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260807 18:17:33.850456 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260807 18:17:34.411748 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:17:35.061038 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260807 18:17:35.606666 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260807 18:17:36.218495 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260807 18:17:36.292574 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260807 18:17:37.550904 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260807 18:17:38.271694 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260807 18:17:38.660761 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260807 18:17:38.703146 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260807 18:17:39.616060 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:17:40.095786 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20260807 18:17:40.569409 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:40.569681 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:40.570390 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:40.572731 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:40.572969 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:40.573108 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:40.573473 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:17:40.667171 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260807 18:17:41.186755 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260807 18:17:41.314041 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260807 18:17:42.398159 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260807 18:17:43.212759 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260807 18:17:43.807296 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260807 18:17:43.841078 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260807 18:17:44.624456 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:17:44.625669 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260807 18:17:45.689848 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260807 18:17:46.044564 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20260807 18:17:46.116397 30224 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:46.116652 30224 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:46.117244 30224 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:46.119285 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:46.119550 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:46.119674 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:46.119935 30224 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:17:46.474433 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260807 18:17:47.143693 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260807 18:17:48.121323 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260807 18:17:48.559836 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260807 18:17:49.008919 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260807 18:17:49.676491 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:17:49.677713 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260807 18:17:50.556406 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260807 18:17:50.755362 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260807 18:17:51.564400 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260807 18:17:52.222456 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260807 18:17:52.917213 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260807 18:17:53.064244 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260807 18:17:54.250770 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260807 18:17:54.791931 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:17:54.868333 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260807 18:17:55.334606 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260807 18:17:55.509191 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20260807 18:17:56.833444 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:17:56.833741 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:17:56.834489 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:17:56.836413 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:17:56.836648 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:17:56.836722 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:17:56.836925 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:17:56.928328 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260807 18:17:57.359303 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260807 18:17:57.725387 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260807 18:17:58.062569 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260807 18:17:59.471798 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260807 18:17:59.894140 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:18:00.045976 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260807 18:18:00.361083 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260807 18:18:00.743475 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260807 18:18:02.202924 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260807 18:18:02.544274 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260807 18:18:02.952332 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260807 18:18:02.970919 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260807 18:18:04.563935 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260807 18:18:05.011219 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:18:05.127853 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260807 18:18:05.398581 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260807 18:18:05.472994 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260807 18:18:06.990334 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260807 18:18:07.771132 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260807 18:18:07.829603 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260807 18:18:07.918073 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260807 18:18:09.673614 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260807 18:18:10.046746 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:18:10.444985 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260807 18:18:10.484601 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260807 18:18:10.550477 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260807 18:18:12.179750 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260807 18:18:12.873396 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260807 18:18:12.878233 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260807 18:18:13.312876 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20260807 18:18:14.522030 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:18:14.522261 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:18:14.522904 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:18:14.524372 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:18:14.524559 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:18:14.524648 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:18:14.524910 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:18:14.619040 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260807 18:18:15.163903 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:18:15.324447 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260807 18:18:15.476892 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260807 18:18:15.689458 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260807 18:18:17.180083 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260807 18:18:17.806304 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260807 18:18:18.005743 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260807 18:18:18.212865 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260807 18:18:19.667366 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260807 18:18:20.290741 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:18:20.291767 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260807 18:18:20.475472 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260807 18:18:20.571542 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260807 18:18:22.120113 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260807 18:18:22.659533 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260807 18:18:23.143183 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260807 18:18:23.302157 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20260807 18:18:23.590090 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:18:23.590292 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:18:23.590866 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:18:23.592794 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:18:23.593011 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:18:23.593125 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:18:23.593454 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:18:24.587883 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260807 18:18:25.058077 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260807 18:18:25.469156 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:18:25.470242 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260807 18:18:25.804386 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260807 18:18:27.312505 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260807 18:18:27.453812 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260807 18:18:27.958328 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260807 18:18:28.291810 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:18:28.468634 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:18:28.613039 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (60 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:18:29.648957 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260807 18:18:29.948428 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260807 18:18:30.577800 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:18:30.578933 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260807 18:18:30.779789 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260807 18:18:32.334661 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260807 18:18:32.392287 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260807 18:18:33.187717 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260807 18:18:33.381989 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260807 18:18:34.677595 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260807 18:18:34.696008 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260807 18:18:35.678869 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:18:35.749800 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260807 18:18:35.981637 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260807 18:18:37.292339 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260807 18:18:37.394196 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260807 18:18:38.472249 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260807 18:18:38.543982 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260807 18:18:39.551163 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260807 18:18:39.888489 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260807 18:18:40.700017 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:18:41.223654 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260807 18:18:41.294627 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260807 18:18:41.954603 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260807 18:18:42.201150 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20260807 18:18:43.250780 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:18:43.251026 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:18:43.251832 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:18:43.253773 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:18:43.254043 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:18:43.254136 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:18:43.254413 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:18:43.686597 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260807 18:18:43.949753 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260807 18:18:44.496811 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260807 18:18:44.811568 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260807 18:18:45.888790 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:18:46.035989 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260807 18:18:46.387728 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260807 18:18:46.862254 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260807 18:18:47.352178 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260807 18:18:48.289207 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260807 18:18:48.903748 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260807 18:18:49.616891 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260807 18:18:49.988054 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260807 18:18:50.747922 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260807 18:18:51.245463 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:18:51.325242 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260807 18:18:52.021371 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260807 18:18:52.399693 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260807 18:18:53.423203 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260807 18:18:53.954980 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260807 18:18:54.598475 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260807 18:18:55.192829 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260807 18:18:56.034982 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260807 18:18:56.198993 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260807 18:18:56.458873 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:18:56.891995 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260807 18:18:57.509778 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260807 18:18:58.351240 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260807 18:18:58.877082 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260807 18:18:59.664048 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260807 18:19:00.127921 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260807 18:19:00.963491 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260807 18:19:01.356932 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260807 18:19:01.472188 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:19:02.165889 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20260807 18:19:02.236362 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:19:02.236660 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:19:02.237385 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:19:02.239662 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:19:02.239849 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:19:02.239989 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:19:02.240265 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:19:02.590505 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260807 18:19:03.672567 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260807 18:19:03.695297 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260807 18:19:04.878786 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260807 18:19:05.412583 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260807 18:19:06.121953 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260807 18:19:06.231526 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260807 18:19:06.543800 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:19:07.359869 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260807 18:19:07.734644 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260807 18:19:08.498171 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260807 18:19:08.586272 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260807 18:19:09.817248 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260807 18:19:10.260622 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260807 18:19:10.796103 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260807 18:19:11.077143 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260807 18:19:11.593890 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:19:12.482826 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260807 18:19:12.799319 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260807 18:19:13.165963 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260807 18:19:13.626066 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260807 18:19:15.098649 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260807 18:19:15.268769 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260807 18:19:15.752231 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260807 18:19:16.181664 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260807 18:19:16.700659 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:19:17.721933 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260807 18:19:18.011843 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260807 18:19:18.334368 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260807 18:19:18.768417 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20260807 18:19:19.466905 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:19:19.467171 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:19:19.467906 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:19:19.469807 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:19:19.470007 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:19:19.470150 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:19:19.470361 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:19:20.462436 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260807 18:19:20.673867 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260807 18:19:20.727764 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260807 18:19:21.453109 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260807 18:19:21.732308 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:19:23.113010 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260807 18:19:23.241511 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260807 18:19:23.243119 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260807 18:19:24.003554 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260807 18:19:25.632984 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260807 18:19:25.884615 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260807 18:19:25.918393 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260807 18:19:26.180931 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260807 18:19:26.838968 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:19:27.871276 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20260807 18:19:28.211095 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:19:28.211313 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:19:28.211990 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:19:28.213745 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:19:28.213943 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:19:28.214051 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:19:28.214334 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:19:28.353464 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260807 18:19:28.383378 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260807 18:19:28.690544 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260807 18:19:28.701155 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (120 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:19:29.751040 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:19:30.380376 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260807 18:19:30.749257 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260807 18:19:30.814954 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260807 18:19:31.131551 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260807 18:19:31.845621 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:19:32.762478 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260807 18:19:33.393203 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260807 18:19:33.398250 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260807 18:19:33.868393 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260807 18:19:35.146502 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260807 18:19:35.773196 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260807 18:19:35.839092 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260807 18:19:36.216544 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260807 18:19:36.937734 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:19:37.726662 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260807 18:19:38.367880 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260807 18:19:38.513065 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260807 18:19:38.541205 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260807 18:19:40.136149 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260807 18:19:40.470194 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20260807 18:19:40.664769 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:19:40.665056 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:19:40.665766 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:19:40.667773 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:19:40.667986 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:19:40.668053 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:19:40.668294 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:19:40.692075 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260807 18:19:40.997283 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260807 18:19:42.048848 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:19:42.674847 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260807 18:19:43.044591 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260807 18:19:43.234997 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260807 18:19:43.579054 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260807 18:19:45.240600 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260807 18:19:45.594889 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260807 18:19:45.618575 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260807 18:19:45.954123 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260807 18:19:47.070725 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:19:47.764945 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260807 18:19:47.992921 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260807 18:19:48.136380 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260807 18:19:48.611933 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260807 18:19:50.224601 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260807 18:19:50.318156 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260807 18:19:50.638739 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260807 18:19:50.951771 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260807 18:19:52.287912 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:19:53.088976 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260807 18:19:53.110937 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260807 18:19:53.157964 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260807 18:19:53.507189 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260807 18:19:55.488759 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260807 18:19:55.748752 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260807 18:19:55.823678 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260807 18:19:55.883021 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260807 18:19:57.501277 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:19:58.087707 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260807 18:19:58.132926 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260807 18:19:58.225824 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260807 18:19:58.237739 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20260807 18:20:00.356740 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:00.357026 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:00.357736 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:00.359390 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:00.359568 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:00.359658 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:00.359968 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:00.439826 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260807 18:20:00.493472 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260807 18:20:00.623157 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260807 18:20:00.830660 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20260807 18:20:02.771878 31896 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:02.772106 31896 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:02.772805 31896 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:02.773897 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:20:02.775079 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:02.775347 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:02.775481 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:02.775815 31896 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:02.837759 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260807 18:20:03.039858 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260807 18:20:03.213286 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260807 18:20:03.307415 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260807 18:20:05.232461 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260807 18:20:05.390149 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260807 18:20:05.713501 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260807 18:20:05.839857 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260807 18:20:07.561563 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260807 18:20:07.784704 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260807 18:20:07.785840 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260807 18:20:08.153890 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260807 18:20:08.230548 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260807 18:20:10.129516 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260807 18:20:10.241796 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260807 18:20:10.666476 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260807 18:20:10.771858 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260807 18:20:12.690747 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260807 18:20:12.862655 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:20:12.942023 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260807 18:20:13.113672 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260807 18:20:13.371662 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260807 18:20:14.807932 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260807 18:20:15.672091 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260807 18:20:15.701958 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260807 18:20:16.051070 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260807 18:20:17.137763 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260807 18:20:17.985381 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:20:18.017823 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260807 18:20:18.189903 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20260807 18:20:18.345813 31950 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:18.346089 31950 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:18.346984 31950 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:18.349149 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:18.349434 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:18.349541 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:18.349774 31950 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:18.560168 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260807 18:20:19.491137 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260807 18:20:20.329577 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260807 18:20:20.556511 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260807 18:20:20.883845 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260807 18:20:22.156352 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260807 18:20:22.765805 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260807 18:20:22.809533 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260807 18:20:23.179190 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260807 18:20:23.180233 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260807 18:20:24.815958 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260807 18:20:25.187914 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260807 18:20:25.418658 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260807 18:20:25.439831 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260807 18:20:27.405433 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260807 18:20:27.771911 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260807 18:20:27.923534 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260807 18:20:27.959515 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260807 18:20:28.344058 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:20:28.789518 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (180 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
I20260807 18:20:29.136917 31950 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:29.137152 31950 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:29.138003 31950 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:29.139928 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:29.140205 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:29.140311 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:29.140589 31950 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:29.948864 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260807 18:20:30.342067 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260807 18:20:30.449501 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260807 18:20:30.498437 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:20:31.035120 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
I20260807 18:20:32.137682 31950 consensus_queue.cc:579] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [LEADER]: Leader has been unable to successfully communicate with peer 4c5d123b5267475ca291a04ab8c14b9c for more than 300 seconds (300.063s)
I20260807 18:20:32.425825 31950 consensus_queue.cc:579] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [LEADER]: Leader has been unable to successfully communicate with peer 4c5d123b5267475ca291a04ab8c14b9c for more than 300 seconds (300.471s)
W20260807 18:20:32.535885 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260807 18:20:32.550192 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260807 18:20:32.970429 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260807 18:20:32.978885 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260807 18:20:33.435577 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20260807 18:20:34.555336 31950 consensus_queue.cc:579] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 [LEADER]: Leader has been unable to successfully communicate with peer b70fc93f83824bcca1b7252f86a22dc7 for more than 300 seconds (300.506s)
I20260807 18:20:34.572857 31973 consensus_queue.cc:579] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 [LEADER]: Leader has been unable to successfully communicate with peer b70fc93f83824bcca1b7252f86a22dc7 for more than 300 seconds (300.533s)
W20260807 18:20:34.683152 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260807 18:20:35.120075 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260807 18:20:35.389890 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260807 18:20:35.591609 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260807 18:20:37.151472 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260807 18:20:37.521522 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260807 18:20:37.925710 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20260807 18:20:37.994692 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:37.994953 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:37.995786 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:37.998072 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:37.998402 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:37.998519 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:37.998819 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:38.029320 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260807 18:20:38.463594 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:20:39.818449 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260807 18:20:40.152550 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260807 18:20:40.312435 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260807 18:20:40.463160 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260807 18:20:42.291997 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260807 18:20:42.764654 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260807 18:20:43.050038 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260807 18:20:43.212935 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260807 18:20:43.510879 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:20:44.595714 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260807 18:20:45.390988 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260807 18:20:45.432978 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260807 18:20:45.649701 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260807 18:20:47.036191 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260807 18:20:47.974653 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260807 18:20:48.096889 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260807 18:20:48.138669 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260807 18:20:48.536978 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:20:49.599268 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260807 18:20:50.187216 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260807 18:20:50.439625 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20260807 18:20:50.483176 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:50.483446 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:50.484185 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:50.486083 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:50.486280 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:50.486351 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:50.486568 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:50.523486 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260807 18:20:52.290725 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260807 18:20:52.515971 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260807 18:20:52.864252 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260807 18:20:52.967175 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20260807 18:20:53.547863 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:20:53.548142 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:20:53.548911 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:20:53.549790 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260807 18:20:53.550881 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:20:53.551074 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:20:53.551149 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:20:53.551364 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:20:54.925511 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260807 18:20:55.051952 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260807 18:20:55.354951 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260807 18:20:55.508068 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260807 18:20:57.542057 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260807 18:20:57.574939 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260807 18:20:57.755813 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260807 18:20:57.860967 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260807 18:20:58.628834 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:20:59.896507 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260807 18:20:59.979233 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260807 18:21:00.103055 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260807 18:21:00.617398 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260807 18:21:02.656787 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260807 18:21:02.796976 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260807 18:21:02.814668 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260807 18:21:03.162879 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260807 18:21:03.670082 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260807 18:21:05.024231 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260807 18:21:05.246333 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260807 18:21:05.276288 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260807 18:21:05.669611 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260807 18:21:07.636797 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260807 18:21:07.661962 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260807 18:21:07.728904 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260807 18:21:08.107462 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260807 18:21:08.735638 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:21:09.846643 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260807 18:21:09.933799 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260807 18:21:10.366286 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260807 18:21:10.419924 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260807 18:21:12.185539 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260807 18:21:12.427394 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260807 18:21:12.715971 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20260807 18:21:12.883736 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:21:12.883939 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:21:12.884516 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:21:12.886475 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:21:12.886819 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:21:12.886917 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:21:12.887209 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:21:13.032583 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260807 18:21:13.783667 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:21:14.574684 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260807 18:21:14.796834 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260807 18:21:14.950589 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260807 18:21:15.537467 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260807 18:21:17.035950 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260807 18:21:17.407724 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260807 18:21:17.904806 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260807 18:21:18.129328 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260807 18:21:18.881873 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:21:19.455147 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260807 18:21:19.860102 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260807 18:21:20.717919 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260807 18:21:20.816573 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260807 18:21:21.720206 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260807 18:21:22.013046 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260807 18:21:23.281785 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260807 18:21:23.339778 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260807 18:21:24.001552 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:21:24.305974 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260807 18:21:24.387353 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260807 18:21:25.684474 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260807 18:21:25.715607 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260807 18:21:26.776773 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260807 18:21:26.810825 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20260807 18:21:27.290110 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:21:27.290335 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:21:27.290947 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:21:27.292755 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:21:27.292956 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:21:27.293020 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:21:27.293229 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:21:28.135449 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260807 18:21:28.258626 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260807 18:21:29.118765 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:21:29.119833 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260807 18:21:29.174548 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260807 18:21:29.658116 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (244 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:21:30.690157 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260807 18:21:30.783798 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260807 18:21:31.379911 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260807 18:21:31.796603 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:21:32.368911 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:21:33.274149 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260807 18:21:33.432415 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260807 18:21:33.678453 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260807 18:21:34.119095 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:21:34.282380 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260807 18:21:35.730024 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260807 18:21:35.792922 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260807 18:21:36.106467 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20260807 18:21:36.686429 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:21:36.688450 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:21:36.689117 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:21:36.690827 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:21:36.690987 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:21:36.691114 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:21:36.691453 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:21:36.794395 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260807 18:21:38.151387 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260807 18:21:38.196800 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260807 18:21:38.670763 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260807 18:21:39.192365 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:21:39.387920 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260807 18:21:40.603076 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260807 18:21:40.624183 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260807 18:21:41.448556 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260807 18:21:41.874773 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260807 18:21:42.848779 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260807 18:21:43.098296 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260807 18:21:43.949007 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260807 18:21:44.385456 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:21:44.479871 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260807 18:21:45.158926 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260807 18:21:45.592506 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20260807 18:21:45.726758 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:21:45.727002 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:21:45.727735 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:21:45.729641 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:21:45.729919 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:21:45.730064 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:21:45.733485 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:21:46.314774 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260807 18:21:46.869669 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260807 18:21:47.700771 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260807 18:21:48.160914 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260807 18:21:48.766016 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260807 18:21:49.557935 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:21:49.558926 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260807 18:21:50.157660 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260807 18:21:50.574716 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260807 18:21:51.409541 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260807 18:21:52.217978 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260807 18:21:52.549945 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260807 18:21:52.973408 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260807 18:21:53.504323 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260807 18:21:54.542750 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260807 18:21:54.566700 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:21:55.134920 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20260807 18:21:55.146549 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:21:55.146821 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:21:55.147469 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:21:55.149910 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:21:55.150074 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:21:55.150143 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:21:55.150391 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:21:55.370858 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260807 18:21:56.039073 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260807 18:21:56.966017 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260807 18:21:57.760696 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260807 18:21:58.087551 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260807 18:21:58.527580 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260807 18:21:59.588511 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:21:59.645272 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260807 18:22:00.419821 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260807 18:22:00.851281 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260807 18:22:01.038203 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260807 18:22:02.036360 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260807 18:22:03.143709 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260807 18:22:03.567183 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260807 18:22:03.984427 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260807 18:22:04.552985 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260807 18:22:04.862661 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:22:05.485404 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260807 18:22:05.898118 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260807 18:22:06.329275 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260807 18:22:07.185141 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260807 18:22:08.118462 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260807 18:22:08.324347 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260807 18:22:08.954648 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260807 18:22:09.484987 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260807 18:22:09.971230 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260807 18:22:10.086745 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:22:10.087028 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:22:10.087733 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:22:10.089762 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:22:10.090014 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:22:10.090117 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:22:10.090411 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:22:10.418232 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260807 18:22:10.932763 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260807 18:22:11.246408 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260807 18:22:12.064731 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260807 18:22:12.625149 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260807 18:22:13.424007 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260807 18:22:13.595459 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260807 18:22:14.747711 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260807 18:22:15.039471 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:22:15.260195 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260807 18:22:15.832998 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260807 18:22:16.098500 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260807 18:22:16.980754 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260807 18:22:17.933970 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260807 18:22:18.318290 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260807 18:22:18.429709 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260807 18:22:19.633481 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260807 18:22:20.059540 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:22:20.537019 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260807 18:22:20.885059 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260807 18:22:20.889869 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20260807 18:22:21.183372 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:22:21.183679 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:22:21.184434 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:22:21.186483 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:22:21.186637 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:22:21.186693 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:22:21.186903 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:22:22.088312 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260807 18:22:23.005442 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260807 18:22:23.390174 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260807 18:22:23.616199 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260807 18:22:24.667996 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260807 18:22:25.091861 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:22:25.526602 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20260807 18:22:25.872730 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:22:25.873047 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:22:25.873960 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:22:25.875801 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:22:25.875998 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:22:25.876092 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:22:25.876533 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:22:25.892254 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260807 18:22:26.054812 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260807 18:22:27.230136 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260807 18:22:27.952606 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260807 18:22:28.250823 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260807 18:22:28.518308 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260807 18:22:29.723538 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260807 18:22:29.742681 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (304 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:22:30.091867 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:22:30.227995 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260807 18:22:30.833796 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260807 18:22:31.064302 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260807 18:22:32.136615 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260807 18:22:32.436069 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260807 18:22:33.254631 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260807 18:22:33.517793 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:22:33.683944 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:22:34.886270 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260807 18:22:34.939534 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260807 18:22:35.119936 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:22:36.077673 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260807 18:22:36.150115 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260807 18:22:37.331089 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260807 18:22:37.348682 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260807 18:22:38.287832 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260807 18:22:38.658720 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20260807 18:22:39.106334 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:22:39.106567 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:22:39.107311 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:22:39.109081 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:22:39.109283 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:22:39.109452 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:22:39.109678 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:22:39.629594 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260807 18:22:40.044734 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260807 18:22:40.223297 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:22:40.683317 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260807 18:22:41.398097 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260807 18:22:42.391639 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260807 18:22:42.428184 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260807 18:22:43.060005 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260807 18:22:43.857739 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260807 18:22:44.715757 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260807 18:22:45.160769 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260807 18:22:45.225246 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:22:45.678512 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260807 18:22:46.257117 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260807 18:22:47.038201 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260807 18:22:47.526626 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260807 18:22:48.165290 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260807 18:22:48.760044 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260807 18:22:49.505815 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260807 18:22:49.967739 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260807 18:22:50.440284 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:22:51.057525 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260807 18:22:51.583719 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260807 18:22:52.229785 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260807 18:22:52.365406 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260807 18:22:53.358979 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20260807 18:22:53.484457 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:22:53.484755 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:22:53.485674 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:22:53.487650 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:22:53.487874 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:22:53.487978 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:22:53.488247 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:22:53.905218 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260807 18:22:54.756078 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260807 18:22:55.116241 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260807 18:22:55.490038 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:22:55.892697 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260807 18:22:56.194412 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260807 18:22:57.102954 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260807 18:22:57.470129 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260807 18:22:58.447328 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260807 18:22:58.829571 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260807 18:22:59.297174 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260807 18:23:00.264580 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260807 18:23:00.679684 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:23:00.977732 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260807 18:23:01.444106 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260807 18:23:01.701504 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260807 18:23:02.715040 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260807 18:23:03.422930 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260807 18:23:03.831069 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260807 18:23:04.127121 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260807 18:23:05.299089 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260807 18:23:05.747680 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:23:05.884586 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260807 18:23:06.279647 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260807 18:23:06.540148 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20260807 18:23:06.978309 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:06.978567 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:06.979193 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:06.980680 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:06.980849 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:06.980918 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:06.981161 31972 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:07.837777 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260807 18:23:08.513031 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260807 18:23:08.999269 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260807 18:23:09.044260 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260807 18:23:10.286237 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260807 18:23:10.815438 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:23:11.139259 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260807 18:23:11.375645 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260807 18:23:11.695466 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260807 18:23:13.234079 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260807 18:23:13.574812 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260807 18:23:13.621138 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260807 18:23:14.074621 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20260807 18:23:14.519701 31972 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:14.519910 31972 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:14.520709 31972 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:14.522862 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:14.523089 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:14.523183 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:14.523442 32029 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:15.396224 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260807 18:23:16.017477 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:23:16.035302 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260807 18:23:16.163839 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260807 18:23:16.747473 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260807 18:23:17.982239 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260807 18:23:18.378988 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260807 18:23:18.587770 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260807 18:23:19.278702 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260807 18:23:20.754294 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260807 18:23:20.872664 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260807 18:23:21.142319 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:23:21.182797 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20260807 18:23:21.470382 32029 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:21.470606 32029 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:21.471333 32029 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:21.473004 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:21.473205 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:21.473272 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:21.473542 32029 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:21.837885 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260807 18:23:23.013612 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260807 18:23:23.370857 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260807 18:23:24.005712 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260807 18:23:24.154278 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260807 18:23:25.664908 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260807 18:23:25.738740 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260807 18:23:26.331055 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:23:26.368301 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260807 18:23:26.822635 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260807 18:23:28.284193 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260807 18:23:28.647142 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260807 18:23:28.942821 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260807 18:23:29.454540 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260807 18:23:29.829656 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (364 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:23:30.936796 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260807 18:23:30.988219 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260807 18:23:31.382825 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:23:31.643924 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260807 18:23:32.039664 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260807 18:23:33.387570 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260807 18:23:33.635504 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260807 18:23:34.084750 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260807 18:23:34.471869 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20260807 18:23:34.772200 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:34.772415 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:34.773118 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:34.775182 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:34.775333 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:34.775390 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:34.775632 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:23:34.958981 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:23:35.905495 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260807 18:23:35.939608 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260807 18:23:36.434792 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:23:36.435853 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260807 18:23:37.289716 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260807 18:23:38.221580 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260807 18:23:38.350185 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260807 18:23:38.851358 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260807 18:23:39.949661 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260807 18:23:40.499068 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260807 18:23:40.953816 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260807 18:23:41.121960 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260807 18:23:41.469762 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:23:42.623181 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20260807 18:23:42.782519 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:42.782888 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:42.783576 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:42.785297 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:42.785542 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:42.785640 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:42.785916 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:42.963311 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260807 18:23:43.493103 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260807 18:23:43.529109 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260807 18:23:45.118110 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260807 18:23:45.702924 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260807 18:23:45.995635 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260807 18:23:46.375882 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260807 18:23:46.735905 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:23:47.850539 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260807 18:23:48.338557 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260807 18:23:48.486527 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260807 18:23:49.029959 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20260807 18:23:50.487300 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:50.487505 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:50.488184 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:50.489671 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:50.489837 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:50.489933 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:50.490207 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:50.577942 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260807 18:23:50.606266 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260807 18:23:50.850180 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260807 18:23:51.341956 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260807 18:23:51.865290 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:23:53.034998 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260807 18:23:53.338690 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260807 18:23:53.342607 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260807 18:23:53.656898 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260807 18:23:55.412575 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260807 18:23:55.834923 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260807 18:23:55.860925 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260807 18:23:55.913640 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260807 18:23:57.023046 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260807 18:23:58.048458 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:23:58.048699 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:23:58.049451 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:23:58.051301 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:23:58.051497 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:23:58.051570 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:23:58.051771 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:23:58.093429 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260807 18:23:58.372212 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260807 18:23:58.375068 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260807 18:23:58.645949 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260807 18:24:00.624595 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260807 18:24:00.951893 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260807 18:24:00.952135 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260807 18:24:01.211927 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260807 18:24:02.192829 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:24:03.326314 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260807 18:24:03.334436 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260807 18:24:03.342746 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260807 18:24:03.767050 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20260807 18:24:05.279452 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:24:05.279711 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:24:05.280354 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:24:05.282496 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:24:05.282725 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:24:05.282840 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:24:05.283136 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:24:05.598930 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260807 18:24:05.725474 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260807 18:24:05.774339 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260807 18:24:06.126420 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260807 18:24:07.225778 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:24:07.997869 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260807 18:24:08.288007 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260807 18:24:08.414870 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260807 18:24:08.456996 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260807 18:24:10.679168 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260807 18:24:10.733718 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260807 18:24:10.789804 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260807 18:24:10.908802 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260807 18:24:12.304966 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:24:12.963546 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260807 18:24:13.069099 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260807 18:24:13.385807 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260807 18:24:13.470981 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20260807 18:24:13.879369 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:24:13.879602 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:24:13.880309 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:24:13.882105 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:24:13.882336 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:24:13.882510 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:24:13.882822 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:24:15.531566 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260807 18:24:15.534438 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260807 18:24:15.949156 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260807 18:24:15.970034 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260807 18:24:17.408578 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:24:17.844815 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260807 18:24:18.129563 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260807 18:24:18.433585 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260807 18:24:18.497941 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260807 18:24:20.386442 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260807 18:24:20.635380 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260807 18:24:20.720690 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260807 18:24:21.236557 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260807 18:24:22.436981 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:24:22.901785 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260807 18:24:23.027424 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260807 18:24:23.267735 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20260807 18:24:24.175491 32030 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:24:24.175803 32030 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:24:24.176488 32030 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:24:24.178761 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:24:24.179030 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:24:24.179113 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:24:24.179344 32030 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:24:24.179869 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260807 18:24:25.452054 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260807 18:24:25.541281 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260807 18:24:25.962898 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260807 18:24:26.847455 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260807 18:24:27.451072 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:24:27.787277 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260807 18:24:27.983982 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260807 18:24:28.256734 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260807 18:24:29.421262 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260807 18:24:29.915025 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (424 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:24:30.384632 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260807 18:24:30.539026 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260807 18:24:30.957326 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260807 18:24:32.030989 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260807 18:24:32.487462 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:24:32.807729 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260807 18:24:33.081158 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260807 18:24:33.318536 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260807 18:24:34.461696 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260807 18:24:35.322268 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260807 18:24:35.459622 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260807 18:24:35.807721 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:24:36.221099 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:24:36.797695 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260807 18:24:37.546684 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:24:37.547662 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260807 18:24:38.080737 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260807 18:24:38.211745 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260807 18:24:39.000579 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260807 18:24:40.024893 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20260807 18:24:40.624760 32053 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:24:40.625029 32053 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:24:40.625840 32053 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:24:40.627833 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:24:40.628026 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:24:40.628122 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:24:40.628345 32053 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:24:40.672955 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260807 18:24:40.848107 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260807 18:24:41.544257 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260807 18:24:42.407444 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260807 18:24:42.674377 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:24:43.176788 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260807 18:24:43.289032 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260807 18:24:44.153115 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260807 18:24:45.159631 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260807 18:24:45.798834 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260807 18:24:46.019733 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260807 18:24:46.521274 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260807 18:24:47.691762 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:24:47.723784 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260807 18:24:48.099713 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260807 18:24:48.645295 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260807 18:24:49.335080 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260807 18:24:50.162679 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260807 18:24:50.214155 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20260807 18:24:50.659334 32073 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:24:50.659600 32073 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:24:50.660429 32073 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:24:50.662210 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:24:50.662523 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:24:50.662607 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:24:50.663303 32074 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:24:51.190169 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260807 18:24:51.828596 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260807 18:24:52.575314 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260807 18:24:52.827214 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:24:52.865381 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260807 18:24:53.802242 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260807 18:24:54.139844 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260807 18:24:54.827999 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260807 18:24:55.461735 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260807 18:24:56.260382 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260807 18:24:56.857637 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260807 18:24:57.081180 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260807 18:24:57.851459 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:24:57.863332 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260807 18:24:58.761533 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260807 18:24:59.220021 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260807 18:24:59.625416 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20260807 18:25:00.261618 32073 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:25:00.261863 32073 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:25:00.262641 32073 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:25:00.264381 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:25:00.264652 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:25:00.264739 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:25:00.265020 32073 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:25:00.326428 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260807 18:25:01.128159 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260807 18:25:01.670099 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260807 18:25:02.005589 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
I20260807 18:25:02.152850 32073 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:25:02.153111 32073 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:25:02.153841 32073 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:25:02.155829 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:25:02.156035 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:25:02.156121 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:25:02.156404 32073 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:25:02.577571 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260807 18:25:02.996166 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20260807 18:25:03.720836 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260807 18:25:04.180006 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260807 18:25:04.702220 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260807 18:25:04.938377 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260807 18:25:06.263916 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260807 18:25:06.579105 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260807 18:25:07.123134 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260807 18:25:07.504990 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260807 18:25:08.040470 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:25:08.860489 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260807 18:25:09.514091 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260807 18:25:09.769560 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260807 18:25:09.973062 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260807 18:25:11.496749 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260807 18:25:12.140816 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260807 18:25:12.398161 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260807 18:25:12.483907 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260807 18:25:13.053932 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:25:13.681550 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260807 18:25:14.484824 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260807 18:25:14.952713 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260807 18:25:14.960914 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260807 18:25:16.000560 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260807 18:25:17.061681 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260807 18:25:17.284984 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260807 18:25:17.387648 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260807 18:25:18.099608 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:25:18.483165 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260807 18:25:19.772018 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260807 18:25:19.936481 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260807 18:25:20.283880 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260807 18:25:21.190558 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20260807 18:25:21.718732 32079 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:25:21.718984 32079 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:25:21.719751 32079 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:25:21.721707 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:25:21.721905 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:25:21.721983 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:25:21.722234 32079 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:25:22.352136 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260807 18:25:22.555755 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260807 18:25:22.704542 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260807 18:25:23.170878 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:25:23.603055 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260807 18:25:24.926862 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260807 18:25:25.120019 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260807 18:25:25.449007 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260807 18:25:25.914985 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260807 18:25:27.563230 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260807 18:25:27.867075 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260807 18:25:27.977097 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260807 18:25:28.326344 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:25:28.327454 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260807 18:25:30.002327 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (484 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:25:30.200213 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260807 18:25:30.460368 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260807 18:25:30.650688 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260807 18:25:30.652607 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260807 18:25:32.578955 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260807 18:25:32.837153 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260807 18:25:32.947022 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260807 18:25:33.220769 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260807 18:25:33.360385 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:25:35.229303 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260807 18:25:35.327870 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260807 18:25:35.334801 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260807 18:25:35.783670 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:25:37.508919 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:25:37.699841 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260807 18:25:37.898391 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260807 18:25:38.010519 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260807 18:25:38.442021 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:25:38.443006 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
I20260807 18:25:38.718847 32079 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:25:38.719076 32079 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:25:38.719732 32079 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:25:38.721771 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:25:38.721992 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:25:38.722069 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:25:38.722281 32079 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:25:40.206020 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260807 18:25:40.260447 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260807 18:25:40.583757 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260807 18:25:40.702500 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260807 18:25:42.424546 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260807 18:25:42.698248 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260807 18:25:43.083158 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260807 18:25:43.098701 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260807 18:25:43.512346 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:25:44.789712 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260807 18:25:44.936746 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260807 18:25:45.687180 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260807 18:25:45.719300 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260807 18:25:47.223718 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260807 18:25:47.290427 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260807 18:25:47.844327 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260807 18:25:48.321250 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260807 18:25:48.569257 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:25:49.651685 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260807 18:25:49.699615 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260807 18:25:50.252750 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260807 18:25:50.970306 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260807 18:25:51.939594 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260807 18:25:52.006878 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260807 18:25:52.804291 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260807 18:25:53.128286 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
I20260807 18:25:53.464479 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:25:53.464707 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:25:53.465451 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:25:53.467259 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:25:53.467437 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:25:53.467525 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:25:53.467752 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:25:53.583714 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:25:54.487377 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260807 18:25:54.522058 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260807 18:25:55.150853 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260807 18:25:55.696491 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260807 18:25:56.926137 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260807 18:25:57.242820 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260807 18:25:57.720846 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260807 18:25:58.169992 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260807 18:25:58.665966 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:25:59.554271 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260807 18:25:59.810145 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260807 18:26:00.075801 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
I20260807 18:26:00.456669 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:26:00.456948 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:26:00.457800 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:26:00.460047 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:26:00.460186 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:26:00.460232 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:26:00.460462 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:26:00.833786 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260807 18:26:02.142658 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260807 18:26:02.178345 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260807 18:26:02.660274 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260807 18:26:03.194689 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260807 18:26:03.722935 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:04.672889 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260807 18:26:04.855484 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260807 18:26:05.031124 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260807 18:26:06.070622 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260807 18:26:06.981775 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260807 18:26:07.531064 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260807 18:26:07.612658 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260807 18:26:08.707474 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260807 18:26:08.977619 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:09.527724 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260807 18:26:09.846402 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260807 18:26:10.052831 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260807 18:26:11.458949 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260807 18:26:12.202423 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260807 18:26:12.303509 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260807 18:26:12.684015 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260807 18:26:13.894179 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260807 18:26:14.121491 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:14.634843 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260807 18:26:14.992406 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260807 18:26:15.478240 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
I20260807 18:26:16.055359 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:26:16.055634 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:26:16.056336 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:26:16.058423 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:26:16.058624 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:26:16.058714 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:26:16.058911 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:26:16.611851 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260807 18:26:17.138236 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260807 18:26:17.156276 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260807 18:26:17.976600 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260807 18:26:19.120573 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260807 18:26:19.290400 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:26:19.720317 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260807 18:26:19.808828 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
I20260807 18:26:20.460928 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:26:20.461252 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:26:20.462002 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:26:20.464177 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:26:20.464421 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:26:20.464526 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:26:20.464862 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:26:20.540733 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260807 18:26:21.727504 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260807 18:26:22.220222 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260807 18:26:22.356810 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260807 18:26:23.019524 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260807 18:26:24.345208 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:24.358251 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260807 18:26:24.820719 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260807 18:26:24.904951 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260807 18:26:25.568717 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260807 18:26:26.731866 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260807 18:26:27.412719 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260807 18:26:27.482195 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260807 18:26:28.274425 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260807 18:26:29.350296 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:29.351356 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260807 18:26:29.617242 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260807 18:26:30.055742 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260807 18:26:30.088452 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (544 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:26:30.989207 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260807 18:26:31.981204 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260807 18:26:32.150847 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260807 18:26:32.190212 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260807 18:26:33.782199 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260807 18:26:34.082819 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260807 18:26:34.368065 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:26:34.898077 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260807 18:26:34.900882 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260807 18:26:36.471450 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260807 18:26:36.723510 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260807 18:26:37.311733 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260807 18:26:37.378839 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
I20260807 18:26:37.841272 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:26:37.841584 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:26:37.842209 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:26:37.844413 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:26:37.844619 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:26:37.844717 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:26:37.845000 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:26:38.816565 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:26:39.155810 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260807 18:26:39.286877 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260807 18:26:39.602730 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:26:39.670290 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260807 18:26:39.852608 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260807 18:26:41.552431 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260807 18:26:41.711452 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260807 18:26:42.245644 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260807 18:26:42.342031 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260807 18:26:44.096498 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260807 18:26:44.247495 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260807 18:26:44.604550 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:26:44.842588 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260807 18:26:44.920082 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260807 18:26:46.538127 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260807 18:26:46.941649 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260807 18:26:47.051159 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260807 18:26:47.424038 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260807 18:26:49.191578 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260807 18:26:49.277472 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260807 18:26:49.588141 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260807 18:26:49.625007 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:26:49.626108 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
I20260807 18:26:51.453219 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:26:51.453467 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:26:51.454262 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:26:51.456122 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:26:51.456357 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:26:51.456446 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:26:51.456727 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:26:51.624599 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260807 18:26:51.763175 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260807 18:26:52.074648 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260807 18:26:52.192708 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260807 18:26:53.981060 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260807 18:26:54.177716 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260807 18:26:54.720090 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:26:54.721390 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260807 18:26:54.800694 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260807 18:26:56.291855 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260807 18:26:56.622699 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260807 18:26:57.327399 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260807 18:26:57.507406 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260807 18:26:59.140965 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260807 18:26:59.178830 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260807 18:26:59.528193 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260807 18:26:59.745661 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:26:59.874277 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
I20260807 18:27:00.599071 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:00.599341 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:00.600047 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:00.601984 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:00.602210 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:00.602291 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:00.602553 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:01.626430 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260807 18:27:01.753675 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260807 18:27:02.195910 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260807 18:27:02.334851 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260807 18:27:04.165447 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260807 18:27:04.225313 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260807 18:27:04.552666 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
I20260807 18:27:04.626740 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:04.626979 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:04.627803 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:04.629653 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:04.629850 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:04.629969 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:04.630261 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:04.738288 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260807 18:27:05.103904 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20260807 18:27:06.434052 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260807 18:27:06.488165 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260807 18:27:07.390446 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260807 18:27:07.455443 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260807 18:27:08.717880 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260807 18:27:08.758607 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260807 18:27:10.047905 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260807 18:27:10.091136 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260807 18:27:10.105296 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:27:11.210309 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260807 18:27:11.240332 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260807 18:27:12.365782 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260807 18:27:12.524001 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260807 18:27:13.975809 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260807 18:27:13.989055 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
I20260807 18:27:14.324364 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:14.324627 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:14.325305 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:14.327484 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:14.327742 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:14.327864 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:14.328137 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:14.822672 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260807 18:27:14.954192 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260807 18:27:15.251606 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:27:16.483307 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260807 18:27:16.567464 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260807 18:27:17.286006 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260807 18:27:17.444741 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260807 18:27:18.903259 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260807 18:27:19.020668 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260807 18:27:19.953689 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260807 18:27:19.971892 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260807 18:27:20.345584 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:27:21.139976 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260807 18:27:21.707659 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260807 18:27:22.326776 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260807 18:27:22.398706 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260807 18:27:23.960690 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260807 18:27:24.411614 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260807 18:27:24.715709 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260807 18:27:24.981091 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260807 18:27:25.459313 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:27:26.417629 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260807 18:27:26.867136 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260807 18:27:27.363636 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260807 18:27:27.484424 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260807 18:27:28.883234 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260807 18:27:29.612074 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260807 18:27:29.852682 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260807 18:27:29.857856 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260807 18:27:30.176389 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (604 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:27:30.515091 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:27:31.339356 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
I20260807 18:27:31.947389 32098 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:31.947551 32098 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:31.948014 32098 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:31.949860 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:31.950058 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:31.950145 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:31.950344 32098 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:32.150204 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260807 18:27:32.190464 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260807 18:27:32.367475 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260807 18:27:33.910298 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260807 18:27:34.611881 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260807 18:27:34.920919 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260807 18:27:35.028828 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260807 18:27:35.547616 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:27:36.601363 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260807 18:27:37.267282 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260807 18:27:37.486737 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260807 18:27:37.567831 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260807 18:27:38.717959 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260807 18:27:39.502259 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260807 18:27:39.944783 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260807 18:27:40.069299 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:27:40.112547 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:27:40.566531 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:27:41.290570 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260807 18:27:41.917282 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260807 18:27:42.572175 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260807 18:27:42.746470 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260807 18:27:43.637530 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260807 18:27:44.402244 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260807 18:27:45.167732 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260807 18:27:45.167940 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260807 18:27:45.687508 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:27:45.733575 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
I20260807 18:27:46.992686 32149 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:46.992946 32149 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:46.993664 32149 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:46.995700 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:46.995898 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:46.995983 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:46.996227 32149 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:47.163465 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260807 18:27:47.450384 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260807 18:27:47.719146 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260807 18:27:48.265997 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260807 18:27:49.912351 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260807 18:27:49.991474 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260807 18:27:50.310309 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260807 18:27:50.720513 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:27:50.913012 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260807 18:27:52.195375 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260807 18:27:52.277107 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260807 18:27:52.875391 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260807 18:27:53.729224 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260807 18:27:54.656409 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260807 18:27:54.720530 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
I20260807 18:27:54.793102 32149 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:27:54.793398 32149 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:27:54.794100 32149 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:27:54.795938 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:27:54.796141 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:27:54.796226 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:27:54.796521 32149 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:27:55.359064 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260807 18:27:55.820387 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:27:56.357448 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260807 18:27:56.893882 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260807 18:27:57.431609 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260807 18:27:57.579393 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260807 18:27:58.843789 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260807 18:27:59.190274 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260807 18:27:59.954707 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260807 18:28:00.084322 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260807 18:28:00.828015 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:28:01.673079 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260807 18:28:01.700168 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260807 18:28:02.012773 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260807 18:28:02.494632 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
I20260807 18:28:03.950506 32154 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:03.950740 32154 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:03.951465 32154 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:03.953505 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:03.953706 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:03.953778 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:03.953989 32154 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:03.975764 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260807 18:28:04.204797 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260807 18:28:04.295353 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260807 18:28:04.983501 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260807 18:28:05.852272 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:28:06.477766 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
I20260807 18:28:06.808017 32154 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:06.808319 32154 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:06.809145 32154 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:06.811226 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:06.811451 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:06.811542 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:06.811808 32154 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:06.817384 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260807 18:28:06.920416 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260807 18:28:07.307487 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260807 18:28:08.993952 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260807 18:28:09.369583 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260807 18:28:09.475426 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260807 18:28:09.695060 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260807 18:28:11.024555 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:28:11.560870 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260807 18:28:11.797829 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260807 18:28:11.835201 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260807 18:28:12.130978 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260807 18:28:14.121435 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260807 18:28:14.324795 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260807 18:28:14.398895 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260807 18:28:14.648048 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260807 18:28:16.125461 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:28:16.679344 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260807 18:28:16.984040 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260807 18:28:17.170954 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260807 18:28:17.178536 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260807 18:28:18.999910 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260807 18:28:19.733899 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260807 18:28:19.800340 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260807 18:28:19.954011 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
I20260807 18:28:19.973094 32154 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:19.973301 32154 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:19.974121 32154 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:19.975996 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:19.976253 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:19.976389 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:19.976694 32154 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:21.460095 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:28:21.482790 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260807 18:28:22.029377 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260807 18:28:22.612680 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260807 18:28:22.646004 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260807 18:28:23.946426 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260807 18:28:24.678643 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260807 18:28:25.017035 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260807 18:28:25.291778 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260807 18:28:26.481487 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:28:26.482577 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260807 18:28:27.038712 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260807 18:28:27.504341 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260807 18:28:27.595044 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260807 18:28:29.165140 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260807 18:28:29.355929 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260807 18:28:29.712245 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260807 18:28:30.147845 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260807 18:28:30.264732 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (664 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:28:31.504766 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:28:31.505885 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260807 18:28:31.582085 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260807 18:28:32.507180 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260807 18:28:32.790623 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260807 18:28:34.109966 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260807 18:28:34.262329 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260807 18:28:34.848500 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260807 18:28:35.681157 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260807 18:28:36.488720 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260807 18:28:36.568408 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260807 18:28:36.690914 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260807 18:28:37.650853 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260807 18:28:38.197156 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260807 18:28:38.950474 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260807 18:28:39.181895 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
I20260807 18:28:39.791147 32154 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:39.791383 32154 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:39.792117 32154 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:39.794473 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:39.794620 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:39.794687 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:39.794942 32154 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:40.379680 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260807 18:28:40.472651 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:28:41.449448 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:28:41.458757 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260807 18:28:41.739959 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:28:41.741046 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260807 18:28:42.760104 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260807 18:28:42.775696 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260807 18:28:43.722208 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260807 18:28:44.016824 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260807 18:28:45.189182 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260807 18:28:45.229664 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260807 18:28:45.991814 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260807 18:28:46.469035 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260807 18:28:46.761880 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:28:47.672683 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260807 18:28:47.917209 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260807 18:28:48.433487 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260807 18:28:49.215065 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260807 18:28:49.932235 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260807 18:28:50.373142 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260807 18:28:50.774618 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
I20260807 18:28:51.054188 32179 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:51.054431 32179 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:51.055224 32179 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:51.057173 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:51.057415 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:51.057544 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:51.057812 32179 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:51.640614 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260807 18:28:51.770511 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:28:52.211712 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260807 18:28:52.820166 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260807 18:28:53.199827 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
I20260807 18:28:53.258733 32180 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:28:53.259006 32180 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:28:53.259706 32180 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:28:53.261644 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:28:53.261852 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:28:53.261956 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:28:53.262223 32180 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:28:54.263871 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260807 18:28:54.656615 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260807 18:28:55.607885 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260807 18:28:55.935285 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260807 18:28:56.622195 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260807 18:28:56.834295 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:28:57.377049 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260807 18:28:58.268311 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260807 18:28:58.557127 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260807 18:28:58.853492 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260807 18:28:59.539324 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260807 18:29:00.801596 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260807 18:29:00.801838 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260807 18:29:01.474409 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260807 18:29:01.860146 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:29:01.861387 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260807 18:29:03.016739 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260807 18:29:03.044634 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260807 18:29:03.986009 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260807 18:29:04.411368 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260807 18:29:05.377089 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260807 18:29:05.769994 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260807 18:29:06.319428 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260807 18:29:07.082790 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:29:07.083786 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260807 18:29:07.663604 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260807 18:29:08.076192 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260807 18:29:08.763341 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
I20260807 18:29:09.398891 32185 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:29:09.399138 32185 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:29:09.399955 32185 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:29:09.402071 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:29:09.402275 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:29:09.402362 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:29:09.402594 32185 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:29:09.596338 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260807 18:29:09.958465 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260807 18:29:10.495237 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260807 18:29:11.372473 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260807 18:29:11.998764 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260807 18:29:12.115821 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:29:12.496052 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260807 18:29:13.041886 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260807 18:29:13.905958 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260807 18:29:14.552907 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260807 18:29:15.185616 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260807 18:29:15.491050 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260807 18:29:16.585611 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260807 18:29:17.024562 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260807 18:29:17.319628 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:29:17.803045 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260807 18:29:17.895697 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260807 18:29:19.065563 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260807 18:29:19.245054 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260807 18:29:20.393246 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260807 18:29:20.479408 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260807 18:29:21.630156 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260807 18:29:21.850610 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260807 18:29:22.453985 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:29:22.959381 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260807 18:29:22.982738 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260807 18:29:23.905750 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260807 18:29:24.423501 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260807 18:29:25.496438 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260807 18:29:25.555753 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
I20260807 18:29:26.523651 32185 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:29:26.523896 32185 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:29:26.524577 32185 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:29:26.526804 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:29:26.527024 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:29:26.527150 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:29:26.527472 32185 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:29:26.600723 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260807 18:29:26.824373 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260807 18:29:27.583662 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:29:28.019006 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260807 18:29:28.262702 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260807 18:29:29.220045 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260807 18:29:29.462306 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260807 18:29:30.355125 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (724 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:29:30.529991 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260807 18:29:30.908908 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260807 18:29:31.802862 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
I20260807 18:29:32.274055 32190 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:29:32.274281 32190 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:29:32.275095 32190 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:29:32.277004 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:29:32.277222 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:29:32.277299 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:29:32.277585 32190 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:29:32.289826 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260807 18:29:32.680154 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:29:33.204075 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260807 18:29:33.678179 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260807 18:29:34.556030 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260807 18:29:34.918037 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260807 18:29:35.692121 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260807 18:29:36.113327 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260807 18:29:37.020625 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260807 18:29:37.376124 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260807 18:29:37.782539 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:29:38.152136 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260807 18:29:38.612802 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260807 18:29:39.237751 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260807 18:29:39.870711 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260807 18:29:40.506683 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260807 18:29:41.134792 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260807 18:29:42.108278 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260807 18:29:42.405395 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.27.4.126:46561) | 127.27.4.126:46561 | UNAVAILABLE
Error from 127.27.4.126:46561: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
FAILED
I20260807 18:29:42.735958 27665 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.4.126:46561, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
W20260807 18:29:42.923535 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:29:42.938868 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260807 18:29:43.390918 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260807 18:29:44.762768 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260807 18:29:44.931118 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260807 18:29:45.428745 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260807 18:29:46.135895 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260807 18:29:47.282441 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260807 18:29:47.386261 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260807 18:29:47.921933 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260807 18:29:48.086851 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:29:48.620157 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
I20260807 18:29:49.337044 32195 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:29:49.337317 32195 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:29:49.337983 32195 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:29:49.339670 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:29:49.339885 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:29:49.339969 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:29:49.340282 32195 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:29:49.575348 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260807 18:29:49.716243 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260807 18:29:50.307266 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260807 18:29:51.115186 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260807 18:29:51.862627 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260807 18:29:52.190326 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260807 18:29:52.728817 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260807 18:29:53.211666 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260807 18:29:53.338274 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260807 18:29:54.475956 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260807 18:29:54.617449 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260807 18:29:55.164723 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260807 18:29:55.856957 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260807 18:29:56.830964 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260807 18:29:56.985536 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
W20260807 18:29:57.755503 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
[New LWP 27668]
[New LWP 27669]
[New LWP 27670]
[New LWP 28278]
[New LWP 28279]
[New LWP 28281]
[New LWP 28282]
[New LWP 32196]
[New LWP 32198]
[New LWP 32199]
[New LWP 32200]
[New LWP 32203]
[New LWP 32204]
[New LWP 32205]
[New LWP 32206]
[New LWP 32207]
W20260807 18:29:58.210556 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260807 18:29:58.319766 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:29:59.064603 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260807 18:29:59.327834 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260807 18:30:00.395396 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260807 18:30:00.767254 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260807 18:30:01.421698 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260807 18:30:01.660979 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260807 18:30:02.699699 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260807 18:30:03.071938 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260807 18:30:03.499631 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260807 18:30:03.962957 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260807 18:30:04.210654 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
I20260807 18:30:04.480116 32195 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:30:04.480396 32195 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:30:04.481134 32195 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:30:04.483093 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:30:04.483309 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:30:04.483385 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:30:04.483667 32195 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:30:05.156082 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260807 18:30:05.472270 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260807 18:30:06.425284 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260807 18:30:06.897552 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260807 18:30:07.487711 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260807 18:30:07.810343 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260807 18:30:08.508539 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260807 18:30:08.853957 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260807 18:30:09.323724 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260807 18:30:09.915751 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260807 18:30:10.415663 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260807 18:30:11.514573 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260807 18:30:11.715200 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260807 18:30:12.191922 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260807 18:30:13.053625 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260807 18:30:13.533533 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260807 18:30:14.280506 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260807 18:30:14.352476 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260807 18:30:14.907783 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260807 18:30:15.617965 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260807 18:30:16.724148 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260807 18:30:17.062357 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260807 18:30:17.649887 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260807 18:30:18.356149 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260807 18:30:18.547981 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260807 18:30:19.276149 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260807 18:30:19.397598 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
I20260807 18:30:20.035038 32195 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:30:20.035264 32195 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:30:20.036000 32195 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:30:20.037355 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:30:20.037541 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:30:20.037631 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:30:20.037885 32195 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:30:20.114439 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260807 18:30:21.070710 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260807 18:30:21.779973 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260807 18:30:22.246871 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260807 18:30:22.714145 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260807 18:30:23.675359 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260807 18:30:23.676513 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260807 18:30:24.182745 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260807 18:30:25.011554 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260807 18:30:25.230195 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
0x00007f812fb47ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260807 18:30:26.069914 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
Id Target Id Frame
* 1 Thread 0x7f8128048d40 (LWP 27665) "all_types-itest" 0x00007f812fb47ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f8123bbd700 (LWP 27668) "all_types-itest" 0x00007f812fb4c32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f81225f4700 (LWP 27669) "kernel-watcher-" 0x00007f812fb47fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f8121df3700 (LWP 27670) "rpc reactor-276" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f81215f2700 (LWP 28278) "rpc reactor-282" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f8120bea700 (LWP 28279) "rpc reactor-282" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f81203e9700 (LWP 28281) "rpc reactor-282" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f811fbe8700 (LWP 28282) "rpc reactor-282" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f811cd3a700 (LWP 32196) "rpc reactor-321" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f811a24c700 (LWP 32198) "rpc reactor-321" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f811b274700 (LWP 32199) "rpc reactor-321" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f811c2b2700 (LWP 32200) "rpc reactor-322" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f811e258700 (LWP 32203) "rpc reactor-322" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f811d53b700 (LWP 32204) "rpc reactor-322" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f811ba9f700 (LWP 32205) "rpc reactor-322" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f8119a35700 (LWP 32206) "rpc reactor-322" 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f811921e700 (LWP 32207) "client-negotiat" 0x00007f812fb47fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f811921e700 (LWP 32207)):
#0 0x00007f812fb47fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f812cd328d9 in kudu::ConditionVariable::WaitFor (this=0x7f812fb47fb9 <pthread_cond_timedwait@@GLIBC_2.3.2+649>, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007f812cfe2e0a in kudu::ThreadPool::DispatchThread (this=0x616000588980) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f812cfe691c in kudu::ThreadPool::CreateThread()::$_2::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
W20260807 18:30:26.662900 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#4 0x00007f812cfe6799 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#5 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000900028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#6 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f8119a35700 (LWP 32206)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001986680, timeout=0.099648731527850032) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001986680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400044f888, flags=8374144, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400044f880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000869768) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f811ba9f700 (LWP 32205)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001986180, timeout=0.099655132507905364) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001986180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400044f688, flags=8373248, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400044f680) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000869628) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f811d53b700 (LWP 32204)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001985c80, timeout=0.043300725519657135) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001985c80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400044f488, flags=8372352, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400044f480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x6110008694e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f811e258700 (LWP 32203)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001985780, timeout=0.099608554504811764) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001985780, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400044f288, flags=8371456, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400044f280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x6110008693a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f811c2b2700 (LWP 32200)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001982580, timeout=0.099370444426313043) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001982580, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400046fe88, flags=8368768, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400046fe80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000868728) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f811b274700 (LWP 32199)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001982080, timeout=0.099375300342217088) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001982080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400046fc88, flags=5566976, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400046fc80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x6110008685e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f811a24c700 (LWP 32198)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001981b80, timeout=0.099317763466387987) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001981b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400046fa88, flags=5566080, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400046fa80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x6110008684a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f811cd3a700 (LWP 32196)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619001981680, timeout=0.09928979747928679) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619001981680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x61400046f888, flags=5565184, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x61400046f880) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000868368) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f811fbe8700 (LWP 28282)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619000055a80, timeout=0.099553664214909077) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619000055a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x614000004288, flags=16256, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000059168) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f81203e9700 (LWP 28281)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619000055580, timeout=0.099580382229760289) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619000055580, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x614000004088, flags=15360, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000059028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f8120bea700 (LWP 28279)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619000055080, timeout=0.099365336820483208) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619000055080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x614000003e88, flags=14464, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000058ee8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
W20260807 18:30:27.453562 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f81215f2700 (LWP 28278)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619000054b80, timeout=0.099030642071738839) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619000054b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x614000003c88, flags=13568, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000058da8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f8121df3700 (LWP 27670)):
#0 0x00007f8128e1b947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f812af0abda in epoll_poll (loop=0x619000052880, timeout=0.099920744309201837) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f812af0e228 in ev_run (loop=0x619000052880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f81335a6e31 in ev::loop_ref::run (this=this@entry=0x614000003488, flags=9088, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f813359f326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f81335a43ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f81335a4229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000051fa8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f81225f4700 (LWP 27669)):
#0 0x00007f812fb47fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f812cd3250e in kudu::ConditionVariable::WaitUntil (this=0x612000009118, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
W20260807 18:30:27.718677 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
#2 0x00007f8140b8502c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f8140b7d4aa in kudu::CountDownLatch::WaitFor (this=0x612000009140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f812ce6845f in kudu::KernelStackWatchdog::RunThread (this=0x612000009040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f812ce6951c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f812ce69399 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f814153dbbd in std::function<void ()>::operator()() const (this=0x611000052228) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f812cfc1eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f8123bbd700 (LWP 27668)):
#0 0x00007f812fb4c32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000039b4fb in __interceptor_waitpid (pid=<optimized out>, status=0x7f8123bbb8b0, options=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f812cfac796 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#3 0x00007f812cfac4c5 in kudu::Subprocess::Wait (this=0x7f8123bbb8b0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#4 0x00007f812ce422be in kudu::PstackWatcher::RunStackDump (argv=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#5 0x00007f812ce419a7 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f812ce40e96 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f812ce3f417 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000005386f9 in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#9 0x0000000000538583 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#10 0x0000000000538503 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#11 0x00000000005384d9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#12 0x0000000000538499 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#13 0x000000000053839b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#14 0x00007f8134488a4f in execute_native_thread_routine () from /tmp/dist-test-taskOVaSOG/build/asan/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#15 0x00007f812fb416db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f8128e1b61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f8128048d40 (LWP 27665)):
#0 0x00007f812fb47ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f812cd321e7 in kudu::ConditionVariable::Wait (this=0x60e000182c90) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
#2 0x00007f8141bf41b9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260807 18:30:28.542343 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260807 18:30:28.791075 27975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111) [suppressed 41 similar messages]
#3 0x00007f81412eaf4b in kudu::Synchronizer::Wait (this=0x7ffc087514e0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f813a58bef3 in kudu::client::KuduClient::Data::ConnectToCluster (this=0x615000052380, client=<optimized out>, deadline=..., creds_policy=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260807 18:30:29.342201 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#5 0x00007f813a50b901 in kudu::client::KuduClientBuilder::Build (this=0x7ffc08751de0, client=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
W20260807 18:30:30.023710 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260807 18:30:30.172755 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260807 18:30:30.455647 28112 heartbeater.cc:644] Failed to heartbeat to 127.27.4.126:46561 (784 consecutive failures): Network error: Failed to ping master at 127.27.4.126:46561: Client connection negotiation failed: client connection to 127.27.4.126:46561: connect: Connection refused (error 111)
#6 0x00007f813aeb1eac in kudu::tools::CreateKuduClient (master_addresses=..., client=<optimized out>, can_see_all_replicas=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
W20260807 18:30:30.879310 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#7 0x00007f8140baac15 in kudu::tools::RemoteKsckCluster::Connect (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:528
#8 0x00007f8140b1e87e in kudu::tools::Ksck::CheckClusterRunning (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f8140b2083f in kudu::tools::Ksck::Run (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f8140b2513d in kudu::tools::Ksck::RunAndPrintResults (this=0x619001985290) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
#11 0x00007f8141b9cdd3 in kudu::ClusterVerifier::RunKsck (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f8141b9c556 in kudu::ClusterVerifier::CheckCluster (this=0x7ffc08753200) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
W20260807 18:30:31.754613 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Couldn't send request to peer 4c5d123b5267475ca291a04ab8c14b9c. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111). This is attempt 1806: this message will repeat every 5th retry.
#13 0x0000000000472628 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x60c00000d000, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x000000000050e27d in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::TestBody (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:804
I20260807 18:30:32.240243 32195 raft_consensus.cc:482] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b70fc93f83824bcca1b7252f86a22dc7)
I20260807 18:30:32.240447 32195 raft_consensus.cc:504] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "879069a857144936b2103cebd0e62e02" member_type: VOTER last_known_addr { host: "127.27.4.66" port: 35099 } } peers { permanent_uuid: "4c5d123b5267475ca291a04ab8c14b9c" member_type: VOTER last_known_addr { host: "127.27.4.67" port: 46537 } } peers { permanent_uuid: "b70fc93f83824bcca1b7252f86a22dc7" member_type: VOTER last_known_addr { host: "127.27.4.65" port: 35371 } }
I20260807 18:30:32.241236 32195 leader_election.cc:288] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537), b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371)
W20260807 18:30:32.246124 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4c5d123b5267475ca291a04ab8c14b9c (127.27.4.67:46537): Network error: Client connection negotiation failed: client connection to 127.27.4.67:46537: connect: Connection refused (error 111)
W20260807 18:30:32.246306 27975 leader_election.cc:334] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111)
I20260807 18:30:32.246398 27975 leader_election.cc:302] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879069a857144936b2103cebd0e62e02; no voters: 4c5d123b5267475ca291a04ab8c14b9c, b70fc93f83824bcca1b7252f86a22dc7
I20260807 18:30:32.246819 32195 raft_consensus.cc:2726] T dbe769098d8d420f97f766bab4ec11e0 P 879069a857144936b2103cebd0e62e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 18:30:32.318198 27975 consensus_peers.cc:487] T b7a1d8dfdd9649978f2c7d8f5ed4d72a P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#15 0x00007f812d81bbfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f812d821a28 "the test body", method=<optimized out>, object=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x60c00000d000, method=<optimized out>, location=location@entry=0x7f812d821a28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f812d8105db in testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f812d810795 in testing::TestInfo::Run (this=0x6110000471c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f812d810cdf in testing::TestSuite::Run (this=0x611000045a00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x611000045a00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f812d81129f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f812d81c13e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f812d8235a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x61600000b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x61600000b480, method=<optimized out>, location=location@entry=0x7f812d8235a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f812d81083d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x0000000000539482 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x0000000000537992 in main (argc=<optimized out>, argv=0x7ffc08753a88) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
W20260807 18:30:32.452234 27975 consensus_peers.cc:487] T 39eaca12a52347f489cae058c0df12d9 P 879069a857144936b2103cebd0e62e02 -> Peer b70fc93f83824bcca1b7252f86a22dc7 (127.27.4.65:35371): Couldn't send request to peer b70fc93f83824bcca1b7252f86a22dc7. Status: Network error: Client connection negotiation failed: client connection to 127.27.4.65:35371: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
************************* END STACKS ***************************
F20260807 18:30:32.454941 27668 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786127432 (unix time) try "date -d @1786127432" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006c11) received by PID 27665 (TID 0x7f8123bbd700) from PID 27665; stack trace: ***
@ 0x7f812ba0aa46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f812fb4c980 (unknown) at ??:0
@ 0x7f8128d38e87 gsignal at ??:0
@ 0x7f8128d3a7f1 abort at ??:0
@ 0x7f812b9fc9ad google::LogMessage::Fail() at ??:0
@ 0x7f812b9ff88a google::LogMessage::SendToLog() at ??:0
@ 0x7f812b9fc4de google::LogMessage::Flush() at ??:0
@ 0x7f812b9fda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x538891 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63
@ 0x538583 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
@ 0x538503 std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x5384d9 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x538499 std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x53839b std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7f8134488a4f execute_native_thread_routine at ??:0
@ 0x7f812fb416db start_thread at ??:0
@ 0x7f8128e1b61f clone at ??:0