Note: This is test shard 6 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/2, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> >
[ RUN ] AllTypesItest/2.TestTimestampPadding
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260811 00:12:14.976560 16057 test_util.cc:268] Using random seed: 1387551228
2026-08-11T00:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T00:12:15Z Disabled control of system clock
I20260811 00:12:15.025710 16057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
/tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.174.126:46603
--webserver_interface=127.15.174.126
--webserver_port=0
--builtin_ntp_servers=127.15.174.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.174.126:46603 with env {}
W20260811 00:12:15.440202 16065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 00:12:15.440660 16065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 00:12:15.440752 16065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 00:12:15.448652 16065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 00:12:15.448832 16065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 00:12:15.448891 16065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 00:12:15.448951 16065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 00:12:15.458923 16065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.174.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.174.126:46603
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.174.126:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.174.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:46:59 UTC on ccycloud.quasar-yjrvid.root.comops.site
build id 4579238
ASAN enabled
I20260811 00:12:15.460975 16065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 00:12:15.463589 16065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 00:12:15.483296 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:15.484244 16074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:15.484361 16072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 00:12:15.485605 16065 server_base.cc:1034] running on GCE node
I20260811 00:12:15.486874 16065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 00:12:15.489284 16065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 00:12:15.490756 16065 hybrid_clock.cc:648] HybridClock initialized: now 1786407135490623 us; error 131 us; skew 500 ppm
I20260811 00:12:15.491319 16065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 00:12:15.501600 16065 webserver.cc:460] Webserver started at http://127.15.174.126:44893/ using document root <none> and password file <none>
I20260811 00:12:15.503041 16065 fs_manager.cc:359] Metadata directory not provided
I20260811 00:12:15.503225 16065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 00:12:15.503788 16065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 00:12:15.508181 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/instance:
uuid: "b6a03aa53aa7405289888d48f06e0e53"
format_stamp: "Formatted at 2026-08-11 00:12:15 on dist-test-slave-p3t5"
I20260811 00:12:15.509184 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal/instance:
uuid: "b6a03aa53aa7405289888d48f06e0e53"
format_stamp: "Formatted at 2026-08-11 00:12:15 on dist-test-slave-p3t5"
I20260811 00:12:15.518311 16065 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260811 00:12:15.525481 16080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:15.528198 16065 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260811 00:12:15.528599 16065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data,/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal
uuid: "b6a03aa53aa7405289888d48f06e0e53"
format_stamp: "Formatted at 2026-08-11 00:12:15 on dist-test-slave-p3t5"
I20260811 00:12:15.528888 16065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 00:12:15.570310 16065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 00:12:15.571503 16065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 00:12:15.571949 16065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 00:12:15.599813 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.174.126:46603 every 8 connection(s)
I20260811 00:12:15.599854 16065 rpc_server.cc:307] RPC server started. Bound to: 127.15.174.126:46603
I20260811 00:12:15.602480 16065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb
I20260811 00:12:15.610635 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:15.611366 16057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu as pid 16065
I20260811 00:12:15.611716 16057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/wal/instance
I20260811 00:12:15.626367 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: Bootstrap starting.
I20260811 00:12:15.632144 16135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:15.634546 16135 log.cc:824] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: Log is configured to *not* fsync() on all Append() calls
I20260811 00:12:15.639935 16135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: No bootstrap required, opened a new log
I20260811 00:12:15.651422 16135 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } }
I20260811 00:12:15.652010 16135 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:15.652226 16135 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a03aa53aa7405289888d48f06e0e53, State: Initialized, Role: FOLLOWER
I20260811 00:12:15.653663 16135 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } }
I20260811 00:12:15.654173 16135 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 00:12:15.654584 16135 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 00:12:15.654872 16135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:15.657532 16135 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } }
I20260811 00:12:15.658679 16135 leader_election.cc:302] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6a03aa53aa7405289888d48f06e0e53; no voters:
I20260811 00:12:15.659479 16135 leader_election.cc:288] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 00:12:15.659896 16140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 00:12:15.660869 16140 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [term 1 LEADER]: Becoming Leader. State: Replica: b6a03aa53aa7405289888d48f06e0e53, State: Running, Role: LEADER
I20260811 00:12:15.661767 16140 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } }
I20260811 00:12:15.662654 16135 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 00:12:15.667699 16142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6a03aa53aa7405289888d48f06e0e53. Latest consensus state: current_term: 1 leader_uuid: "b6a03aa53aa7405289888d48f06e0e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } } }
I20260811 00:12:15.668303 16142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [sys.catalog]: This master's current role is: LEADER
I20260811 00:12:15.669667 16141 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6a03aa53aa7405289888d48f06e0e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6a03aa53aa7405289888d48f06e0e53" member_type: VOTER last_known_addr { host: "127.15.174.126" port: 46603 } } }
I20260811 00:12:15.669991 16141 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53 [sys.catalog]: This master's current role is: LEADER
I20260811 00:12:15.671105 16151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 00:12:15.681113 16151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 00:12:15.697202 16151 catalog_manager.cc:1347] Generated new cluster ID: 0bee90efbf3e4adfbb069bedc252476e
I20260811 00:12:15.697403 16151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 00:12:15.714738 16151 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 00:12:15.716315 16151 catalog_manager.cc:1504] Loading token signing keys...
I20260811 00:12:15.723857 16151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: Generated new TSK 0
I20260811 00:12:15.724936 16151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 00:12:15.739212 16057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
/tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.174.65:0
--local_ip_for_outbound_sockets=127.15.174.65
--webserver_interface=127.15.174.65
--webserver_port=0
--tserver_master_addrs=127.15.174.126:46603
--builtin_ntp_servers=127.15.174.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260811 00:12:16.130870 16161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 00:12:16.131181 16161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 00:12:16.131369 16161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 00:12:16.139108 16161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 00:12:16.139313 16161 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 00:12:16.139412 16161 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 00:12:16.139585 16161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.174.65
I20260811 00:12:16.149340 16161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.174.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.174.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.174.65
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.174.126:46603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.174.65
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:46:59 UTC on ccycloud.quasar-yjrvid.root.comops.site
build id 4579238
ASAN enabled
I20260811 00:12:16.150827 16161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 00:12:16.153007 16161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 00:12:16.168994 16170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:16.169003 16168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:16.168994 16167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 00:12:16.169939 16161 server_base.cc:1034] running on GCE node
I20260811 00:12:16.170643 16161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 00:12:16.171818 16161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 00:12:16.173086 16161 hybrid_clock.cc:648] HybridClock initialized: now 1786407136172988 us; error 63 us; skew 500 ppm
I20260811 00:12:16.173521 16161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 00:12:16.176139 16161 webserver.cc:460] Webserver started at http://127.15.174.65:38587/ using document root <none> and password file <none>
I20260811 00:12:16.176920 16161 fs_manager.cc:359] Metadata directory not provided
I20260811 00:12:16.177066 16161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 00:12:16.177438 16161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 00:12:16.180364 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/instance:
uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.181317 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal/instance:
uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.187808 16161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 00:12:16.191797 16176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.193524 16161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 00:12:16.193833 16161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal
uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.194043 16161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 00:12:16.206183 16161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 00:12:16.207095 16161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 00:12:16.207366 16161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 00:12:16.208456 16161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 00:12:16.210299 16161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 00:12:16.210378 16161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.210538 16161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 00:12:16.210589 16161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.249192 16161 rpc_server.cc:307] RPC server started. Bound to: 127.15.174.65:37601
I20260811 00:12:16.249315 16298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.174.65:37601 every 8 connection(s)
I20260811 00:12:16.250670 16161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/info.pb
I20260811 00:12:16.255336 16057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu as pid 16161
I20260811 00:12:16.255563 16057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/wal/instance
I20260811 00:12:16.259632 16057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
/tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.174.66:0
--local_ip_for_outbound_sockets=127.15.174.66
--webserver_interface=127.15.174.66
--webserver_port=0
--tserver_master_addrs=127.15.174.126:46603
--builtin_ntp_servers=127.15.174.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 00:12:16.269068 16299 heartbeater.cc:347] Connected to a master server at 127.15.174.126:46603
I20260811 00:12:16.269600 16299 heartbeater.cc:464] Registering TS with master...
I20260811 00:12:16.270781 16299 heartbeater.cc:511] Master 127.15.174.126:46603 requested a full tablet report, sending...
I20260811 00:12:16.275637 16099 ts_manager.cc:194] Registered new tserver with Master: b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601)
I20260811 00:12:16.279155 16099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.174.65:44301
W20260811 00:12:16.654652 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 00:12:16.654990 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 00:12:16.655087 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 00:12:16.663157 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 00:12:16.663357 16303 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 00:12:16.663420 16303 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 00:12:16.663512 16303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.174.66
I20260811 00:12:16.674058 16303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.174.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.174.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.174.66
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.174.126:46603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.174.66
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:46:59 UTC on ccycloud.quasar-yjrvid.root.comops.site
build id 4579238
ASAN enabled
I20260811 00:12:16.675483 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 00:12:16.677511 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 00:12:16.693789 16308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:16.693923 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:16.694314 16309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 00:12:16.695109 16303 server_base.cc:1034] running on GCE node
I20260811 00:12:16.696048 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 00:12:16.697165 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 00:12:16.698580 16303 hybrid_clock.cc:648] HybridClock initialized: now 1786407136698417 us; error 71 us; skew 500 ppm
I20260811 00:12:16.699101 16303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 00:12:16.702066 16303 webserver.cc:460] Webserver started at http://127.15.174.66:46737/ using document root <none> and password file <none>
I20260811 00:12:16.702914 16303 fs_manager.cc:359] Metadata directory not provided
I20260811 00:12:16.703087 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 00:12:16.703451 16303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 00:12:16.706604 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/instance:
uuid: "2d01f4514a644a2ea81db4c9f506e19d"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.707582 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal/instance:
uuid: "2d01f4514a644a2ea81db4c9f506e19d"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.714241 16303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 00:12:16.718855 16318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.720453 16303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260811 00:12:16.720698 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal
uuid: "2d01f4514a644a2ea81db4c9f506e19d"
format_stamp: "Formatted at 2026-08-11 00:12:16 on dist-test-slave-p3t5"
I20260811 00:12:16.720984 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 00:12:16.739640 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 00:12:16.740638 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 00:12:16.740993 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 00:12:16.742169 16303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 00:12:16.744093 16303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 00:12:16.744218 16303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.744380 16303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 00:12:16.744463 16303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:16.790608 16303 rpc_server.cc:307] RPC server started. Bound to: 127.15.174.66:46197
I20260811 00:12:16.790652 16444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.174.66:46197 every 8 connection(s)
I20260811 00:12:16.792060 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/info.pb
I20260811 00:12:16.798748 16057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu as pid 16303
I20260811 00:12:16.798998 16057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/wal/instance
I20260811 00:12:16.803119 16057 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
/tmp/dist-test-taskdqxoSS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.174.67:0
--local_ip_for_outbound_sockets=127.15.174.67
--webserver_interface=127.15.174.67
--webserver_port=0
--tserver_master_addrs=127.15.174.126:46603
--builtin_ntp_servers=127.15.174.84:40145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260811 00:12:16.809801 16445 heartbeater.cc:347] Connected to a master server at 127.15.174.126:46603
I20260811 00:12:16.810168 16445 heartbeater.cc:464] Registering TS with master...
I20260811 00:12:16.811026 16445 heartbeater.cc:511] Master 127.15.174.126:46603 requested a full tablet report, sending...
I20260811 00:12:16.812701 16099 ts_manager.cc:194] Registered new tserver with Master: 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
I20260811 00:12:16.814160 16099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.174.66:57593
W20260811 00:12:17.203958 16449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 00:12:17.204285 16449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 00:12:17.204380 16449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 00:12:17.212358 16449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 00:12:17.212517 16449 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260811 00:12:17.212612 16449 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260811 00:12:17.212739 16449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.174.67
I20260811 00:12:17.222522 16449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.174.84:40145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.174.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.174.67
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.15.174.126:46603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.174.67
--log_dir=/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:46:59 UTC on ccycloud.quasar-yjrvid.root.comops.site
build id 4579238
ASAN enabled
I20260811 00:12:17.223891 16449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 00:12:17.226182 16449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 00:12:17.243095 16454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:17.242995 16455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 00:12:17.243038 16457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 00:12:17.244825 16449 server_base.cc:1034] running on GCE node
I20260811 00:12:17.245682 16449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 00:12:17.246794 16449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 00:12:17.248243 16449 hybrid_clock.cc:648] HybridClock initialized: now 1786407137248145 us; error 46 us; skew 500 ppm
I20260811 00:12:17.248639 16449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 00:12:17.254523 16449 webserver.cc:460] Webserver started at http://127.15.174.67:45995/ using document root <none> and password file <none>
I20260811 00:12:17.255342 16449 fs_manager.cc:359] Metadata directory not provided
I20260811 00:12:17.255512 16449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 00:12:17.255867 16449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 00:12:17.258931 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/instance:
uuid: "29d3790ed6ae4e509b332ca347d6f5de"
format_stamp: "Formatted at 2026-08-11 00:12:17 on dist-test-slave-p3t5"
I20260811 00:12:17.259820 16449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal/instance:
uuid: "29d3790ed6ae4e509b332ca347d6f5de"
format_stamp: "Formatted at 2026-08-11 00:12:17 on dist-test-slave-p3t5"
I20260811 00:12:17.266120 16449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 00:12:17.269996 16463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:17.271643 16449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 00:12:17.271857 16449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal
uuid: "29d3790ed6ae4e509b332ca347d6f5de"
format_stamp: "Formatted at 2026-08-11 00:12:17 on dist-test-slave-p3t5"
I20260811 00:12:17.272132 16449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 00:12:17.284938 16299 heartbeater.cc:503] Master 127.15.174.126:46603 was elected leader, sending a full tablet report...
I20260811 00:12:17.288492 16449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 00:12:17.289351 16449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 00:12:17.289685 16449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 00:12:17.290704 16449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 00:12:17.292193 16449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 00:12:17.292269 16449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:17.292358 16449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 00:12:17.292404 16449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 00:12:17.342116 16449 rpc_server.cc:307] RPC server started. Bound to: 127.15.174.67:45711
I20260811 00:12:17.342192 16585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.174.67:45711 every 8 connection(s)
I20260811 00:12:17.343977 16449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb
I20260811 00:12:17.349502 16057 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdqxoSS/build/asan/bin/kudu as pid 16449
I20260811 00:12:17.349714 16057 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/wal/instance
I20260811 00:12:17.360731 16586 heartbeater.cc:347] Connected to a master server at 127.15.174.126:46603
I20260811 00:12:17.361075 16586 heartbeater.cc:464] Registering TS with master...
I20260811 00:12:17.362110 16586 heartbeater.cc:511] Master 127.15.174.126:46603 requested a full tablet report, sending...
I20260811 00:12:17.363812 16099 ts_manager.cc:194] Registered new tserver with Master: 29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711)
I20260811 00:12:17.364744 16099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.174.67:37567
I20260811 00:12:17.367826 16057 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 00:12:17.395835 16099 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39566:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 00:12:17.405967 16099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 00:12:17.450521 16515 tablet_service.cc:1467] Processing CreateTablet for tablet f4dfa5ed7be8439a88f32ac097f9c56d (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 00:12:17.449383 16514 tablet_service.cc:1467] Processing CreateTablet for tablet 8de88d670149435a8031520dfd947788 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 00:12:17.452507 16514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de88d670149435a8031520dfd947788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.449497 16513 tablet_service.cc:1467] Processing CreateTablet for tablet 1c04612d65bb419893570a7a89c44b60 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 00:12:17.455928 16228 tablet_service.cc:1467] Processing CreateTablet for tablet 8de88d670149435a8031520dfd947788 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 00:12:17.456090 16229 tablet_service.cc:1467] Processing CreateTablet for tablet f4dfa5ed7be8439a88f32ac097f9c56d (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 00:12:17.456720 16370 tablet_service.cc:1467] Processing CreateTablet for tablet 1c04612d65bb419893570a7a89c44b60 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 00:12:17.458141 16228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de88d670149435a8031520dfd947788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.458261 16227 tablet_service.cc:1467] Processing CreateTablet for tablet 1c04612d65bb419893570a7a89c44b60 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20260811 00:12:17.458470 16370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c04612d65bb419893570a7a89c44b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.458804 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c04612d65bb419893570a7a89c44b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.459299 16229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dfa5ed7be8439a88f32ac097f9c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.460808 16513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c04612d65bb419893570a7a89c44b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.456754 16373 tablet_service.cc:1467] Processing CreateTablet for tablet f4dfa5ed7be8439a88f32ac097f9c56d (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION VALUES < 715827882
I20260811 00:12:17.462126 16373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dfa5ed7be8439a88f32ac097f9c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.457139 16371 tablet_service.cc:1467] Processing CreateTablet for tablet 8de88d670149435a8031520dfd947788 (DEFAULT_TABLE table=all-types-table [id=425c4212e69f4d9f84cac21ddaec2b65]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20260811 00:12:17.463189 16371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de88d670149435a8031520dfd947788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.465085 16515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4dfa5ed7be8439a88f32ac097f9c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 00:12:17.491789 16606 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: Bootstrap starting.
I20260811 00:12:17.494601 16608 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Bootstrap starting.
I20260811 00:12:17.494815 16607 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: Bootstrap starting.
I20260811 00:12:17.496356 16606 tablet_bootstrap.cc:654] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.498466 16606 log.cc:824] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: Log is configured to *not* fsync() on all Append() calls
I20260811 00:12:17.499083 16608 tablet_bootstrap.cc:654] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.499080 16607 tablet_bootstrap.cc:654] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.500576 16607 log.cc:824] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: Log is configured to *not* fsync() on all Append() calls
I20260811 00:12:17.500576 16608 log.cc:824] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Log is configured to *not* fsync() on all Append() calls
I20260811 00:12:17.504760 16607 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: No bootstrap required, opened a new log
I20260811 00:12:17.504760 16608 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: No bootstrap required, opened a new log
I20260811 00:12:17.505082 16608 ts_tablet_manager.cc:1397] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260811 00:12:17.505082 16607 ts_tablet_manager.cc:1397] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260811 00:12:17.508104 16606 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: No bootstrap required, opened a new log
I20260811 00:12:17.508477 16606 ts_tablet_manager.cc:1397] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260811 00:12:17.515905 16608 raft_consensus.cc:348] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.516526 16608 raft_consensus.cc:374] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.516642 16608 raft_consensus.cc:729] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d3790ed6ae4e509b332ca347d6f5de, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.517441 16608 consensus_queue.cc:260] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.517434 16606 raft_consensus.cc:348] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.517956 16606 raft_consensus.cc:374] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.518095 16606 raft_consensus.cc:729] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d01f4514a644a2ea81db4c9f506e19d, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.518548 16607 raft_consensus.cc:348] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.519022 16607 raft_consensus.cc:374] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.519126 16586 heartbeater.cc:503] Master 127.15.174.126:46603 was elected leader, sending a full tablet report...
I20260811 00:12:17.519130 16607 raft_consensus.cc:729] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cd95c75ac84415bbae31cced0aa1c0, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.519742 16608 ts_tablet_manager.cc:1428] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260811 00:12:17.519775 16607 consensus_queue.cc:260] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.520239 16608 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de: Bootstrap starting.
I20260811 00:12:17.521883 16607 ts_tablet_manager.cc:1428] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260811 00:12:17.522506 16607 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: Bootstrap starting.
I20260811 00:12:17.522910 16608 tablet_bootstrap.cc:654] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.524806 16607 tablet_bootstrap.cc:654] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.531760 16608 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de: No bootstrap required, opened a new log
I20260811 00:12:17.531824 16607 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: No bootstrap required, opened a new log
I20260811 00:12:17.531963 16607 ts_tablet_manager.cc:1397] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260811 00:12:17.531963 16608 ts_tablet_manager.cc:1397] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260811 00:12:17.533049 16607 raft_consensus.cc:348] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.533049 16608 raft_consensus.cc:348] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.533308 16607 raft_consensus.cc:374] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.533332 16608 raft_consensus.cc:374] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.533411 16607 raft_consensus.cc:729] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cd95c75ac84415bbae31cced0aa1c0, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.533435 16608 raft_consensus.cc:729] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d3790ed6ae4e509b332ca347d6f5de, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.533440 16606 consensus_queue.cc:260] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.533725 16608 consensus_queue.cc:260] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.533864 16607 consensus_queue.cc:260] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.534363 16608 ts_tablet_manager.cc:1428] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 00:12:17.534510 16607 ts_tablet_manager.cc:1428] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260811 00:12:17.534514 16617 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 00:12:17.534665 16608 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de: Bootstrap starting.
I20260811 00:12:17.534770 16607 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0: Bootstrap starting.
I20260811 00:12:17.535035 16617 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.536648 16607 tablet_bootstrap.cc:654] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.536648 16608 tablet_bootstrap.cc:654] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.551254 16617 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711)
I20260811 00:12:17.553066 16445 heartbeater.cc:503] Master 127.15.174.126:46603 was elected leader, sending a full tablet report...
I20260811 00:12:17.555190 16606 ts_tablet_manager.cc:1428] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d: Time spent starting tablet: real 0.046s user 0.013s sys 0.006s
I20260811 00:12:17.555657 16606 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: Bootstrap starting.
I20260811 00:12:17.556134 16608 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de: No bootstrap required, opened a new log
I20260811 00:12:17.556375 16608 ts_tablet_manager.cc:1397] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.013s
I20260811 00:12:17.557308 16608 raft_consensus.cc:348] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.557492 16606 tablet_bootstrap.cc:654] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.557633 16608 raft_consensus.cc:374] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.557737 16608 raft_consensus.cc:729] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d3790ed6ae4e509b332ca347d6f5de, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.558053 16608 consensus_queue.cc:260] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.558650 16607 tablet_bootstrap.cc:492] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0: No bootstrap required, opened a new log
I20260811 00:12:17.558807 16608 ts_tablet_manager.cc:1428] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260811 00:12:17.558825 16607 ts_tablet_manager.cc:1397] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.013s
I20260811 00:12:17.559659 16607 raft_consensus.cc:348] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.559912 16607 raft_consensus.cc:374] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.560062 16607 raft_consensus.cc:729] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cd95c75ac84415bbae31cced0aa1c0, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.560326 16607 consensus_queue.cc:260] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.560912 16607 ts_tablet_manager.cc:1428] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 00:12:17.564051 16612 raft_consensus.cc:482] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 00:12:17.564476 16612 raft_consensus.cc:504] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.567047 16252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de88d670149435a8031520dfd947788" candidate_uuid: "2d01f4514a644a2ea81db4c9f506e19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" is_pre_election: true
I20260811 00:12:17.567497 16606 tablet_bootstrap.cc:492] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: No bootstrap required, opened a new log
I20260811 00:12:17.567664 16606 ts_tablet_manager.cc:1397] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260811 00:12:17.568041 16252 raft_consensus.cc:2454] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d01f4514a644a2ea81db4c9f506e19d in term 0.
I20260811 00:12:17.568988 16606 raft_consensus.cc:348] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.569351 16606 raft_consensus.cc:374] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.569481 16606 raft_consensus.cc:729] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d01f4514a644a2ea81db4c9f506e19d, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.569852 16606 consensus_queue.cc:260] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.570079 16320 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0; no voters:
I20260811 00:12:17.570712 16617 raft_consensus.cc:2781] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 00:12:17.570959 16617 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 00:12:17.571138 16617 raft_consensus.cc:3037] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.570725 16606 ts_tablet_manager.cc:1428] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260811 00:12:17.571919 16606 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: Bootstrap starting.
I20260811 00:12:17.573918 16612 leader_election.cc:288] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
I20260811 00:12:17.574746 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de88d670149435a8031520dfd947788" candidate_uuid: "2d01f4514a644a2ea81db4c9f506e19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d3790ed6ae4e509b332ca347d6f5de" is_pre_election: true
I20260811 00:12:17.575412 16538 raft_consensus.cc:2454] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d01f4514a644a2ea81db4c9f506e19d in term 0.
I20260811 00:12:17.578696 16606 tablet_bootstrap.cc:654] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: Neither blocks nor log segments found. Creating new log.
I20260811 00:12:17.579628 16617 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.580860 16252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de88d670149435a8031520dfd947788" candidate_uuid: "2d01f4514a644a2ea81db4c9f506e19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
I20260811 00:12:17.581117 16252 raft_consensus.cc:3037] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.581488 16393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c04612d65bb419893570a7a89c44b60" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01f4514a644a2ea81db4c9f506e19d" is_pre_election: true
I20260811 00:12:17.582041 16393 raft_consensus.cc:2454] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 0.
I20260811 00:12:17.583004 16466 leader_election.cc:302] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de, 2d01f4514a644a2ea81db4c9f506e19d; no voters:
I20260811 00:12:17.583534 16612 raft_consensus.cc:2781] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 00:12:17.583698 16612 raft_consensus.cc:482] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 00:12:17.583806 16612 raft_consensus.cc:3037] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.584656 16252 raft_consensus.cc:2454] T 8de88d670149435a8031520dfd947788 P b7cd95c75ac84415bbae31cced0aa1c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d01f4514a644a2ea81db4c9f506e19d in term 1.
I20260811 00:12:17.586107 16617 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [CANDIDATE]: Term 1 election: Requested vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711)
I20260811 00:12:17.586722 16612 raft_consensus.cc:504] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.586957 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de88d670149435a8031520dfd947788" candidate_uuid: "2d01f4514a644a2ea81db4c9f506e19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d3790ed6ae4e509b332ca347d6f5de"
I20260811 00:12:17.587129 16538 raft_consensus.cc:3037] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.587728 16320 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0; no voters:
I20260811 00:12:17.588825 16393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c04612d65bb419893570a7a89c44b60" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01f4514a644a2ea81db4c9f506e19d"
I20260811 00:12:17.589035 16393 raft_consensus.cc:3037] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.589747 16612 leader_election.cc:288] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 election: Requested vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
I20260811 00:12:17.590477 16538 raft_consensus.cc:2454] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d01f4514a644a2ea81db4c9f506e19d in term 1.
I20260811 00:12:17.590381 16251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c04612d65bb419893570a7a89c44b60" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
I20260811 00:12:17.590663 16251 raft_consensus.cc:3037] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.591811 16606 tablet_bootstrap.cc:492] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: No bootstrap required, opened a new log
I20260811 00:12:17.592006 16606 ts_tablet_manager.cc:1397] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.010s
I20260811 00:12:17.592722 16251 raft_consensus.cc:2454] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 1.
I20260811 00:12:17.593183 16393 raft_consensus.cc:2454] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 1.
I20260811 00:12:17.593214 16465 leader_election.cc:302] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de, b7cd95c75ac84415bbae31cced0aa1c0; no voters:
I20260811 00:12:17.593195 16252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c04612d65bb419893570a7a89c44b60" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" is_pre_election: true
I20260811 00:12:17.593314 16606 raft_consensus.cc:348] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.593598 16606 raft_consensus.cc:374] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 00:12:17.593719 16612 raft_consensus.cc:2781] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 00:12:17.593758 16606 raft_consensus.cc:729] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d01f4514a644a2ea81db4c9f506e19d, State: Initialized, Role: FOLLOWER
I20260811 00:12:17.593969 16252 raft_consensus.cc:2362] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 1. Re-sending same reply.
I20260811 00:12:17.594098 16606 consensus_queue.cc:260] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.594390 16617 raft_consensus.cc:2781] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 00:12:17.594671 16617 raft_consensus.cc:686] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [term 1 LEADER]: Becoming Leader. State: Replica: 2d01f4514a644a2ea81db4c9f506e19d, State: Running, Role: LEADER
I20260811 00:12:17.594868 16606 ts_tablet_manager.cc:1428] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260811 00:12:17.595376 16612 raft_consensus.cc:686] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 LEADER]: Becoming Leader. State: Replica: 29d3790ed6ae4e509b332ca347d6f5de, State: Running, Role: LEADER
I20260811 00:12:17.595376 16617 consensus_queue.cc:237] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
W20260811 00:12:17.596289 16587 tablet.cc:2369] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 00:12:17.596221 16612 consensus_queue.cc:237] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:17.610118 16099 catalog_manager.cc:5495] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67). New cstate: current_term: 1 leader_uuid: "29d3790ed6ae4e509b332ca347d6f5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } health_report { overall_health: HEALTHY } } }
I20260811 00:12:17.609571 16098 catalog_manager.cc:5495] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66). New cstate: current_term: 1 leader_uuid: "2d01f4514a644a2ea81db4c9f506e19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } health_report { overall_health: UNKNOWN } } }
I20260811 00:12:17.693243 16612 raft_consensus.cc:482] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 00:12:17.693532 16612 raft_consensus.cc:504] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.694316 16612 leader_election.cc:288] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
I20260811 00:12:17.695008 16393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dfa5ed7be8439a88f32ac097f9c56d" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01f4514a644a2ea81db4c9f506e19d" is_pre_election: true
I20260811 00:12:17.695073 16252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dfa5ed7be8439a88f32ac097f9c56d" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" is_pre_election: true
I20260811 00:12:17.695305 16393 raft_consensus.cc:2454] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 0.
I20260811 00:12:17.695446 16252 raft_consensus.cc:2454] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 0.
I20260811 00:12:17.695771 16466 leader_election.cc:302] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de, 2d01f4514a644a2ea81db4c9f506e19d; no voters:
I20260811 00:12:17.696110 16612 raft_consensus.cc:2781] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 00:12:17.696198 16612 raft_consensus.cc:482] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 00:12:17.696276 16612 raft_consensus.cc:3037] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.698391 16612 raft_consensus.cc:504] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.699043 16612 leader_election.cc:288] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 election: Requested vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
I20260811 00:12:17.699666 16393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dfa5ed7be8439a88f32ac097f9c56d" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d01f4514a644a2ea81db4c9f506e19d"
I20260811 00:12:17.699666 16252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4dfa5ed7be8439a88f32ac097f9c56d" candidate_uuid: "29d3790ed6ae4e509b332ca347d6f5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cd95c75ac84415bbae31cced0aa1c0"
I20260811 00:12:17.699970 16252 raft_consensus.cc:3037] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.699973 16393 raft_consensus.cc:3037] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 0 FOLLOWER]: Advancing to term 1
I20260811 00:12:17.703323 16252 raft_consensus.cc:2454] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 1.
I20260811 00:12:17.703325 16393 raft_consensus.cc:2454] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d3790ed6ae4e509b332ca347d6f5de in term 1.
I20260811 00:12:17.703851 16465 leader_election.cc:302] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de, b7cd95c75ac84415bbae31cced0aa1c0; no voters:
I20260811 00:12:17.704198 16612 raft_consensus.cc:2781] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 00:12:17.704423 16612 raft_consensus.cc:686] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [term 1 LEADER]: Becoming Leader. State: Replica: 29d3790ed6ae4e509b332ca347d6f5de, State: Running, Role: LEADER
I20260811 00:12:17.704732 16612 consensus_queue.cc:237] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } }
I20260811 00:12:17.707863 16098 catalog_manager.cc:5495] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67). New cstate: current_term: 1 leader_uuid: "29d3790ed6ae4e509b332ca347d6f5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } health_report { overall_health: UNKNOWN } } }
W20260811 00:12:17.753990 16300 tablet.cc:2369] T 1c04612d65bb419893570a7a89c44b60 P b7cd95c75ac84415bbae31cced0aa1c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 00:12:17.794513 16446 tablet.cc:2369] T 1c04612d65bb419893570a7a89c44b60 P 2d01f4514a644a2ea81db4c9f506e19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 00:12:17.996515 16617 consensus_queue.cc:1035] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:18.005257 16617 consensus_queue.cc:1035] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:18.081086 16612 consensus_queue.cc:1035] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:18.087358 16612 consensus_queue.cc:1035] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:18.099574 16612 consensus_queue.cc:1035] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:18.106760 16612 consensus_queue.cc:1035] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 00:12:19.603430 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling FlushMRSOp(f4dfa5ed7be8439a88f32ac097f9c56d): perf score=1.000000
W20260811 00:12:19.924304 16593 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.174.67:45711, user_credentials={real_user=slave}} blocked reactor thread for 85032.9us
I20260811 00:12:20.009567 16300 maintenance_manager.cc:419] P b7cd95c75ac84415bbae31cced0aa1c0: Scheduling FlushMRSOp(f4dfa5ed7be8439a88f32ac097f9c56d): perf score=1.000000
I20260811 00:12:20.050175 16446 maintenance_manager.cc:419] P 2d01f4514a644a2ea81db4c9f506e19d: Scheduling FlushMRSOp(f4dfa5ed7be8439a88f32ac097f9c56d): perf score=1.937278
W20260811 00:12:20.226845 16326 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-1/data/data (499298304 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 00:12:20.245730 16181 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-0/data/data (465739776 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 00:12:20.314353 16181 tablet_replica_mm_ops.cc:214] T f4dfa5ed7be8439a88f32ac097f9c56d P b7cd95c75ac84415bbae31cced0aa1c0: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet f4dfa5ed7be8439a88f32ac097f9c56d: unable to open output file for column decimal64_val DECIMAL(18, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in f4dfa5ed7be8439a88f32ac097f9c56d's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 00:12:20.314556 16181 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet f4dfa5ed7be8439a88f32ac097f9c56d: unable to open output file for column decimal64_val DECIMAL(18, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in f4dfa5ed7be8439a88f32ac097f9c56d's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f2321af39ad google::LogMessage::Fail() at ??:0
@ 0x7f2321af688a google::LogMessage::SendToLog() at ??:0
@ 0x7f2321af34de google::LogMessage::Flush() at ??:0
@ 0x7f2321af4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2330ecce7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f23245e1090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f23245e3e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f23245e3ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f232473f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f232474291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2324742799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f232471deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f23279f76db start_thread at ??:0
@ 0x7f232098d61f clone at ??:0
W20260811 00:12:21.113178 16465 connection.cc:535] client connection to 127.15.174.65:37601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 00:12:21.113749 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 00:12:21.134191 16320 connection.cc:535] client connection to 127.15.174.65:37601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 00:12:21.134521 16320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 00:12:21.140504 16706 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 00:12:21.130456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.174.65:37601 (local address 127.15.174.67:48713)
0811 00:12:21.138674 (+ 8218us) negotiation.cc:107] Waiting for socket to connect
0811 00:12:21.139060 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":8015,"threads_started":1}
W20260811 00:12:21.142055 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20260811 00:12:21.142273 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20260811 00:12:21.147887 16593 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.174.67:45711, user_credentials={real_user=slave}} blocked reactor thread for 144766us
W20260811 00:12:21.163458 16320 consensus_peers.cc:487] T 8de88d670149435a8031520dfd947788 P 2d01f4514a644a2ea81db4c9f506e19d -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 00:12:21.407702 16697 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56194 (ReqId={client: 89eb4bbd48c4430991241ce0592ebd50, seq_no=3, attempt_no=0}) took 1086 ms. Trace:
I20260811 00:12:21.412453 16697 rpcz_store.cc:276] 0811 00:12:20.321391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 00:12:20.321486 (+ 95us) service_pool.cc:227] Handling call
0811 00:12:21.407630 (+1086144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0811 00:12:20.342410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8de88d670149435a8031520dfd947788
0811 00:12:20.342735 (+ 325us) write_op.cc:432] Acquiring schema lock in shared mode
0811 00:12:20.342743 (+ 8us) write_op.cc:435] Acquired schema lock
0811 00:12:20.342746 (+ 3us) tablet.cc:660] Decoding operations
0811 00:12:20.519161 (+176415us) write_op.cc:620] Acquiring the partition lock for write op
0811 00:12:20.519183 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0811 00:12:20.519191 (+ 8us) tablet.cc:683] Acquiring locks for 6286 operations
0811 00:12:20.680891 (+161700us) tablet.cc:699] Row locks acquired
0811 00:12:20.680897 (+ 6us) write_op.cc:260] PREPARE: finished
0811 00:12:20.680987 (+ 90us) write_op.cc:270] Start()
0811 00:12:20.681041 (+ 54us) write_op.cc:276] Timestamp: P: 1786407140680916 usec, L: 0
0811 00:12:20.681045 (+ 4us) op_driver.cc:331] REPLICATION: starting
0811 00:12:20.699464 (+ 18419us) log.cc:842] Serialized 993826 byte log entry
0811 00:12:20.777511 (+ 78047us) op_driver.cc:444] REPLICATION: finished
0811 00:12:20.778153 (+ 642us) write_op.cc:301] APPLY: starting
0811 00:12:20.778178 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0811 00:12:20.796697 (+ 18519us) tablet.cc:1368] finished BulkCheckPresence
0811 00:12:20.796791 (+ 94us) tablet.cc:1370] starting ApplyRowOperation cycle
0811 00:12:21.365071 (+568280us) tablet.cc:1381] finished ApplyRowOperation cycle
0811 00:12:21.375812 (+ 10741us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0811 00:12:21.375823 (+ 11us) write_op.cc:312] APPLY: finished
0811 00:12:21.402659 (+ 26836us) log.cc:842] Serialized 37738 byte log entry
0811 00:12:21.402818 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0811 00:12:21.407019 (+ 4201us) write_op.cc:454] Released schema lock
0811 00:12:21.407388 (+ 369us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":541,"mutex_wait_us":26,"num_ops":6286,"prepare.queue_time_us":17967,"prepare.run_cpu_time_us":221628,"prepare.run_wall_time_us":439391,"raft.queue_time_us":2200,"raft.run_cpu_time_us":4228,"raft.run_wall_time_us":20787,"replication_time_us":96424,"spinlock_wait_cycles":896,"thread_start_us":104914,"threads_started":4,"wal-append.queue_time_us":41555}]]}
W20260811 00:12:21.543823 16591 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.174.66:46197, user_credentials={real_user=slave}} blocked reactor thread for 116353us
W20260811 00:12:22.192680 16326 tablet_replica_mm_ops.cc:214] T f4dfa5ed7be8439a88f32ac097f9c56d P 2d01f4514a644a2ea81db4c9f506e19d: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in f4dfa5ed7be8439a88f32ac097f9c56d's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 00:12:22.193212 16326 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in f4dfa5ed7be8439a88f32ac097f9c56d's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7ff3c0b819ad google::LogMessage::Fail() at ??:0
@ 0x7ff3c0b8488a google::LogMessage::SendToLog() at ??:0
@ 0x7ff3c0b814de google::LogMessage::Flush() at ??:0
@ 0x7ff3c0b82a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff3cff5ae7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7ff3c366f090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7ff3c3671e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7ff3c3671ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff3c37cd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff3c37d091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff3c37d0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff3c37abeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff3c6a856db start_thread at ??:0
@ 0x7ff3bfa1b61f clone at ??:0
W20260811 00:12:22.316854 16593 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.174.67:45711, user_credentials={real_user=slave}} blocked reactor thread for 58434us
I20260811 00:12:22.502388 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260811 00:12:22.543128 16466 connection.cc:535] client connection to 127.15.174.66:46197 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 00:12:22.544982 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260811 00:12:22.545575 16763 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 00:12:22.463089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.174.66:46197 (local address 127.0.0.1:56200)
0811 00:12:22.475688 (+ 12599us) negotiation.cc:107] Waiting for socket to connect
0811 00:12:22.475714 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 00:12:22.476280 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 00:12:22.543612 (+ 67332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12369,"thread_start_us":12780,"threads_started":1}
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:22.565865 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:22.586347 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 00:12:22.675501 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: FlushMRSOp(f4dfa5ed7be8439a88f32ac097f9c56d) complete. Timing: real 3.071s user 1.306s sys 0.222s Metrics: {"bytes_written":436486,"cfile_init":1,"compiler_manager_pool.queue_time_us":30562,"compiler_manager_pool.run_cpu_time_us":1106527,"compiler_manager_pool.run_wall_time_us":2310234,"dirs.queue_time_us":816,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":157,"drs_written":1,"lbm_read_time_us":154,"lbm_reads_lt_1ms":4,"lbm_write_time_us":30879,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":84,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":34988,"threads_started":3,"wal-append.queue_time_us":415}
I20260811 00:12:22.677331 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling FlushMRSOp(1c04612d65bb419893570a7a89c44b60): perf score=1.937278
I20260811 00:12:22.817106 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:22.863492 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:12:23.193426 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:23.226397 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:12:23.238806 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:23.239063 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:23.239745 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:23.241648 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:12:23.241817 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:12:23.241940 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:23.242211 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:12:23.301164 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 00:12:23.409046 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: FlushMRSOp(1c04612d65bb419893570a7a89c44b60) complete. Timing: real 0.731s user 0.621s sys 0.107s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":419,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":921,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3390,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":594,"threads_started":2,"wal-append.queue_time_us":452}
I20260811 00:12:23.410157 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling FlushMRSOp(8de88d670149435a8031520dfd947788): perf score=1.937278
W20260811 00:12:23.440802 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 00:12:23.665091 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:23.710915 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:12:24.085991 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: FlushMRSOp(8de88d670149435a8031520dfd947788) complete. Timing: real 0.676s user 0.630s sys 0.045s Metrics: {"bytes_written":472622,"cfile_init":1,"dirs.queue_time_us":439,"dirs.run_cpu_time_us":324,"dirs.run_wall_time_us":1602,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3194,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":532,"threads_started":2,"wal-append.queue_time_us":372}
I20260811 00:12:24.086829 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling UndoDeltaBlockGCOp(1c04612d65bb419893570a7a89c44b60): 63923 bytes on disk
I20260811 00:12:24.087834 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: UndoDeltaBlockGCOp(1c04612d65bb419893570a7a89c44b60) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20260811 00:12:24.088397 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling UndoDeltaBlockGCOp(8de88d670149435a8031520dfd947788): 63425 bytes on disk
I20260811 00:12:24.088871 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: UndoDeltaBlockGCOp(8de88d670149435a8031520dfd947788) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260811 00:12:24.089361 16587 maintenance_manager.cc:419] P 29d3790ed6ae4e509b332ca347d6f5de: Scheduling UndoDeltaBlockGCOp(f4dfa5ed7be8439a88f32ac097f9c56d): 62923 bytes on disk
I20260811 00:12:24.089859 16469 maintenance_manager.cc:643] P 29d3790ed6ae4e509b332ca347d6f5de: UndoDeltaBlockGCOp(f4dfa5ed7be8439a88f32ac097f9c56d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20260811 00:12:24.302708 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:24.333348 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:24.852301 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260811 00:12:24.980119 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260811 00:12:25.079669 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:25.079882 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:25.080462 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:25.082267 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:12:25.082425 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:12:25.082566 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:25.082813 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 00:12:25.177008 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:25.209225 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:25.671563 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260811 00:12:26.147452 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:12:26.148535 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260811 00:12:26.293901 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:26.326436 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:26.918285 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260811 00:12:27.226873 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:27.227192 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:27.227871 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:27.229607 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:12:27.230161 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:12:27.230247 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:27.230473 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:12:27.348423 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260811 00:12:27.420725 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:27.453480 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:28.202788 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260811 00:12:28.503862 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260811 00:12:28.549165 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:28.596860 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:29.257644 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260811 00:12:29.688239 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:29.721295 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:30.007839 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260811 00:12:30.406385 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:30.406600 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:30.407321 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:30.408928 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:12:30.409089 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:12:30.409210 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:30.409438 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:12:30.462620 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 00:12:30.809823 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:30.841461 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:30.910745 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260811 00:12:31.286296 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20260811 00:12:31.518090 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 00:12:31.930819 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:31.962571 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:32.356536 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260811 00:12:32.534309 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:32.534759 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:32.535908 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:32.537436 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:12:32.537688 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:12:32.537784 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:32.538022 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 00:12:33.056417 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:33.089593 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:33.264875 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260811 00:12:33.411818 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260811 00:12:33.822223 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 00:12:34.182914 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:34.214921 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:34.677471 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260811 00:12:35.297353 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:35.331030 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:35.669091 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260811 00:12:35.681092 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 00:12:36.415864 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:36.448614 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:36.464154 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:12:36.465091 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260811 00:12:37.240265 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260811 00:12:37.540462 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:37.572961 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:38.168042 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260811 00:12:38.225430 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 00:12:38.672092 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:38.720618 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:39.127710 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260811 00:12:39.583117 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260811 00:12:39.814148 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:39.847239 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:40.232600 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260811 00:12:40.934628 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:40.967437 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:41.043465 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260811 00:12:41.616724 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:12:41.631426 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260811 00:12:41.856541 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260811 00:12:42.050882 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:42.083436 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:12:42.403788 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:42.404031 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:42.404816 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:42.406454 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:12:42.406797 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:12:42.406939 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:42.407171 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:12:42.636158 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 00:12:43.168691 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:43.200078 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:43.541438 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260811 00:12:44.236021 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260811 00:12:44.294298 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:44.328392 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:44.586895 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260811 00:12:44.991353 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260811 00:12:45.419652 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:45.451823 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:45.780375 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260811 00:12:46.545778 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:46.584021 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:46.754323 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:12:46.916973 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 00:12:47.256538 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260811 00:12:47.409991 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 00:12:47.792861 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:47.869199 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:48.441254 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260811 00:12:48.956393 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:48.991029 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:49.379570 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260811 00:12:49.612345 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260811 00:12:49.682703 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 00:12:50.089408 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:50.120333 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:51.007014 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 00:12:51.206317 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:51.239389 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:51.489135 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260811 00:12:51.825496 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:12:52.085549 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260811 00:12:52.238654 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:12:52.238870 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:12:52.239682 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:12:52.241510 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:12:52.242169 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:12:52.242436 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:12:52.242738 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 00:12:52.326409 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:52.359117 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:52.440057 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260811 00:12:53.101189 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260811 00:12:53.448477 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:53.479511 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:54.113770 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260811 00:12:54.573937 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:54.607367 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:54.920168 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260811 00:12:54.927217 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 00:12:55.697924 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:55.730584 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:55.803514 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 00:12:56.813947 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:56.844856 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:56.882838 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:12:56.883899 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260811 00:12:57.347906 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260811 00:12:57.372664 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260811 00:12:57.934396 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:57.966225 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:58.335583 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 00:12:59.058641 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:12:59.095311 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:12:59.564901 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260811 00:12:59.654861 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 00:12:59.852808 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260811 00:13:00.198860 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:00.238977 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:00.738094 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260811 00:13:01.338181 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:01.374140 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:02.033162 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:13:02.034679 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260811 00:13:02.269855 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260811 00:13:02.271938 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260811 00:13:02.474447 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:02.510010 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:03.537672 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 00:13:03.637100 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:03.673673 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:04.393994 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260811 00:13:04.712354 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 00:13:04.778719 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:04.817449 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:05.126057 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260811 00:13:05.920886 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:05.958890 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:06.299860 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260811 00:13:06.908531 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260811 00:13:07.051067 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:07.083721 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:07.184258 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:13:07.277365 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260811 00:13:07.414913 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260811 00:13:08.170749 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:08.203406 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:08.610404 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 00:13:09.282842 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 00:13:09.292106 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:09.324380 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:09.901963 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260811 00:13:09.967590 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260811 00:13:10.144387 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:13:10.144645 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:13:10.145305 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:13:10.146908 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:13:10.147116 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:13:10.147226 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:13:10.147547 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 00:13:10.417155 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:10.457737 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:10.888921 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 00:13:11.545399 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:11.583514 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:11.705231 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260811 00:13:12.125990 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260811 00:13:12.190591 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:13:12.263696 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260811 00:13:12.680375 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:12.713559 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:13.125355 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 00:13:13.802350 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:13.834261 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:14.153362 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260811 00:13:14.422757 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260811 00:13:14.536787 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260811 00:13:14.927301 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:14.965988 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:15.653251 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260811 00:13:16.070571 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:16.107806 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:16.597465 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 00:13:16.707160 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260811 00:13:17.119684 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260811 00:13:17.200264 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260811 00:13:17.205219 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:17.234287 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:13:18.334640 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:18.365450 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:18.408859 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 00:13:19.005187 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260811 00:13:19.359614 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 00:13:19.450858 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:19.483664 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:19.558955 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260811 00:13:20.583104 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:20.619088 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:20.931598 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260811 00:13:21.731527 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:21.767155 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:21.888911 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260811 00:13:21.930707 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260811 00:13:21.939129 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260811 00:13:22.342597 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20260811 00:13:22.860006 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:22.892527 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:23.480305 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260811 00:13:23.988153 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:24.020373 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:24.301153 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260811 00:13:24.424254 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260811 00:13:24.539426 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260811 00:13:25.120026 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:25.154675 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:25.964329 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260811 00:13:26.261746 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:26.300415 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:26.766770 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260811 00:13:26.849804 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260811 00:13:27.006464 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260811 00:13:27.351291 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260811 00:13:27.398172 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:27.430124 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:28.384121 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 00:13:28.523041 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:28.554749 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:13:28.802484 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:13:28.802652 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:13:28.803326 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:13:28.804754 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:13:28.804958 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:13:28.805058 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:13:28.805299 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:13:29.352252 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260811 00:13:29.402781 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260811 00:13:29.427263 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260811 00:13:29.653146 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:29.686961 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:13:30.789479 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:30.824271 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:30.943641 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260811 00:13:31.470228 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 00:13:31.733497 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260811 00:13:31.920039 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260811 00:13:31.934075 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:31.955189 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:32.453166 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20260811 00:13:33.044545 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:33.076681 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:33.394515 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 00:13:33.692788 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20260811 00:13:34.167619 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:34.199086 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:34.359408 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260811 00:13:34.378712 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260811 00:13:35.311661 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:35.344897 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:35.883383 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260811 00:13:35.942512 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 00:13:36.449358 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:36.482615 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:36.790151 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260811 00:13:37.136247 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260811 00:13:37.530918 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260811 00:13:37.586066 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:37.627808 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:38.335549 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260811 00:13:38.492769 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 00:13:38.720679 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:38.753101 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:39.054937 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 00:13:39.069218 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:13:39.069420 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:13:39.070163 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:13:39.072119 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:13:39.072396 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:13:39.072503 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:13:39.072765 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:13:39.843604 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260811 00:13:39.849503 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:39.882637 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:13:40.770762 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:13:40.770983 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:13:40.771752 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:13:40.773362 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:13:40.773985 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:13:40.774163 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:13:40.774466 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:13:40.841006 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 00:13:40.989773 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:41.021330 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:41.227537 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260811 00:13:41.577570 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260811 00:13:42.122941 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:42.157861 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:42.345474 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260811 00:13:42.700044 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20260811 00:13:43.267269 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:43.304731 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:43.404027 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260811 00:13:43.887938 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 00:13:44.132902 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 00:13:44.398011 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:44.431393 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:44.864184 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260811 00:13:45.518766 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:45.550813 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:45.975342 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 00:13:46.520658 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260811 00:13:46.565690 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260811 00:13:46.657876 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:46.691463 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:47.337292 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260811 00:13:47.702344 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260811 00:13:47.798712 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:47.833441 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:48.315392 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 00:13:48.937484 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:48.984370 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:49.066542 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260811 00:13:49.322957 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260811 00:13:50.024230 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260811 00:13:50.074659 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:50.105125 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:50.783644 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 00:13:51.194327 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:51.225595 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:51.664983 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260811 00:13:51.929665 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20260811 00:13:52.336573 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:52.365504 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:52.505316 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260811 00:13:52.716624 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260811 00:13:52.842355 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:13:52.842576 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:13:52.843266 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:13:52.845084 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:13:52.845306 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:13:52.845428 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:13:52.845779 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:13:53.264142 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260811 00:13:53.462158 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:53.499030 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:54.169322 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260811 00:13:54.420256 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20260811 00:13:54.604360 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:54.646103 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:55.171842 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260811 00:13:55.590950 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 00:13:55.736078 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:55.768081 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:56.715155 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 00:13:56.859825 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:56.892905 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:57.069037 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260811 00:13:57.774840 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:13:57.775998 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260811 00:13:58.002074 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:58.036295 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:58.082572 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 00:13:59.136240 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:13:59.172446 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:13:59.371901 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260811 00:13:59.440646 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260811 00:14:00.236724 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260811 00:14:00.262459 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:00.298089 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:00.372373 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260811 00:14:01.107254 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:01.107468 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:01.108238 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:01.110023 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:14:01.110183 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:14:01.110301 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:01.110536 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 00:14:01.384294 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:01.416164 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:02.002193 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260811 00:14:02.085356 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260811 00:14:02.512470 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:02.544452 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:02.750026 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260811 00:14:02.817507 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:14:02.822968 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 00:14:03.661221 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:03.691864 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:04.348877 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 00:14:04.566067 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260811 00:14:04.781380 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:04.813498 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:05.381697 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260811 00:14:05.483152 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260811 00:14:05.907474 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:05.939065 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:06.908560 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 00:14:07.031915 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:07.063342 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:07.128647 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 00:14:07.812436 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260811 00:14:08.020002 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:14:08.099999 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20260811 00:14:08.150691 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:08.181910 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:14:09.275601 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260811 00:14:09.308712 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:09.317972 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:09.485989 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260811 00:14:10.098588 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260811 00:14:10.292898 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 00:14:10.409482 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:10.442868 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:14:11.528285 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:11.559201 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:11.768138 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260811 00:14:12.207985 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260811 00:14:12.648170 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:12.681347 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:12.805668 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260811 00:14:12.868707 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260811 00:14:13.146607 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260811 00:14:13.774488 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:13.807905 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 00:14:13.941529 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:13.941802 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:13.942402 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:13.944047 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:14:13.944293 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:14:13.944406 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:13.944630 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:14.274098 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 00:14:14.858918 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260811 00:14:14.911417 16513 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
b6a03aa53aa7405289888d48f06e0e53 | 127.15.174.126:46603 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.174.84:40145 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+-------------+----------+----------------+-----------------
29d3790ed6ae4e509b332ca347d6f5de | 127.15.174.67:45711 | HEALTHY | <none> | 2 | 0
2d01f4514a644a2ea81db4c9f506e19d | 127.15.174.66:46197 | UNAVAILABLE | <none> | n/a | n/a
b7cd95c75ac84415bbae31cced0aa1c0 | 127.15.174.65:37601 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.15.174.66:46197: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.15.174.65:37601: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+---------------------
flush_threshold_mb | 1 | runtime,experimental | 127.15.174.67:45711
ipki_server_key_size | 768 | experimental | 127.15.174.67:45711
local_ip_for_outbound_sockets | 127.15.174.67 | experimental | 127.15.174.67:45711
never_fsync | true | unsafe,advanced | 127.15.174.67:45711
openssl_security_level_override | 0 | unsafe,hidden | 127.15.174.67:45711
rpc_server_allow_ephemeral_ports | true | unsafe | 127.15.174.67:45711
server_dump_info_format | pb | hidden | 127.15.174.67:45711
server_dump_info_path | /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.174.67:45711
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.15.174.67:45711
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+---------------------
builtin_ntp_servers | 127.15.174.84:40145 | 127.15.174.67:45711
time_source | builtin | 127.15.174.67:45711
Version Summary
Version | Servers
-----------------+----------------------------------------------------------
1.18.0-SNAPSHOT | master@127.15.174.126:46603, tserver@127.15.174.67:45711
Tablet Summary
Tablet f4dfa5ed7be8439a88f32ac097f9c56d of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
All reported replicas are:
A = 29d3790ed6ae4e509b332ca347d6f5de
B = b7cd95c75ac84415bbae31cced0aa1c0
C = 2d01f4514a644a2ea81db4c9f506e19d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 8de88d670149435a8031520dfd947788 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable [LEADER]
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
Tablet 1c04612d65bb419893570a7a89c44b60 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): TS unavailable
2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): TS unavailable
29d3790ed6ae4e509b332ca347d6f5de (127.15.174.67:45711): RUNNING [LEADER]
All reported replicas are:
A = b7cd95c75ac84415bbae31cced0aa1c0
B = 2d01f4514a644a2ea81db4c9f506e19d
C = 29d3790ed6ae4e509b332ca347d6f5de
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 00:14:14.944782 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260811 00:14:15.396766 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260811 00:14:15.569446 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260811 00:14:15.777593 16088 maintenance_manager.cc:419] P b6a03aa53aa7405289888d48f06e0e53: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033366
W20260811 00:14:15.780602 16085 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskdqxoSS/test-tmp/all_types-itest.5.AllTypesItest_2.TestTimestampPadding.1786407134871368-16057-0/minicluster-data/master-0/data/data (462553088 bytes available vs 518351011 bytes reserved) (error 28)
W20260811 00:14:15.781426 16085 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P b6a03aa53aa7405289888d48f06e0e53: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260811 00:14:15.781539 16085 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f8c752109ad google::LogMessage::Fail() at ??:0
@ 0x7f8c7521388a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c752104de google::LogMessage::Flush() at ??:0
@ 0x7f8c75211a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c845e9e7d kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f8c77cfe090 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f8c77d00e53 kudu::MaintenanceManager::RunSchedulerThread()::$_3::operator()() at ??:0
@ 0x7f8c77d00ce9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8c77e5c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8c77e5f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8c77e5f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8c77e3aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c7b1146db start_thread at ??:0
@ 0x7f8c740aa61f clone at ??:0
W20260811 00:14:16.026548 19589 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 00:14:15.948185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.174.126:46603 (local address 127.0.0.1:45636)
0811 00:14:15.948990 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0811 00:14:15.949016 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 00:14:15.949183 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 00:14:16.026179 (+ 76996us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":472,"threads_started":1}
W20260811 00:14:16.390183 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:14:16.800298 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 00:14:17.285069 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 00:14:17.763391 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260811 00:14:18.176858 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:14:18.177903 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 00:14:19.237603 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 00:14:19.862172 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 00:14:20.205441 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260811 00:14:20.616020 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 00:14:21.623965 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 00:14:22.381415 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 00:14:22.672560 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260811 00:14:22.965299 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 00:14:23.225894 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:14:24.292718 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 00:14:24.831748 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20260811 00:14:24.856299 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:24.856564 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:24.857520 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:24.859174 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:14:24.859351 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:14:24.859534 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:24.859818 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:25.046662 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260811 00:14:25.607417 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 00:14:26.555727 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 00:14:27.289740 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 00:14:27.414561 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260811 00:14:28.272193 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:14:28.343425 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 00:14:29.243358 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 00:14:29.885108 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260811 00:14:29.905208 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20260811 00:14:29.907676 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:29.907944 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:29.908548 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:29.910072 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:14:29.910283 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:14:29.910406 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:29.910646 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:31.057379 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 00:14:31.887765 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 00:14:32.578249 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 00:14:32.583504 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260811 00:14:33.466290 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:14:33.467377 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 00:14:34.284348 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 00:14:34.835242 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20260811 00:14:34.957185 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:34.957389 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:34.958233 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:34.959894 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:14:34.960039 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:14:34.960140 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:34.960350 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:35.396770 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260811 00:14:36.039296 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 00:14:36.743852 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 00:14:37.245491 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 00:14:37.703504 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260811 00:14:38.527237 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:14:38.528347 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 00:14:39.132531 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20260811 00:14:39.628403 16710 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:39.628666 16710 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:39.629382 16710 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:39.631278 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:14:39.631533 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:14:39.631677 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:39.631948 16710 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:40.080654 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 00:14:40.399833 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260811 00:14:41.273379 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 00:14:41.658759 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 00:14:42.579439 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 00:14:42.917284 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260811 00:14:43.581825 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:14:43.808056 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 00:14:43.960793 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 00:14:45.191497 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 00:14:45.286468 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260811 00:14:46.669835 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 00:14:46.729758 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 00:14:47.705019 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 00:14:47.932435 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260811 00:14:48.737607 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:14:49.058081 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 00:14:49.123268 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 00:14:49.944329 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 00:14:50.632771 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260811 00:14:51.296151 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 00:14:51.485385 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20260811 00:14:52.520882 19603 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:14:52.521103 19603 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:14:52.521770 19603 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:14:52.523768 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:14:52.523927 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:14:52.524067 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:14:52.524271 19603 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:14:52.543332 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 00:14:52.970808 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260811 00:14:53.831934 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:14:53.833106 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 00:14:54.246902 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 00:14:55.015167 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 00:14:55.496488 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260811 00:14:56.185012 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 00:14:56.950798 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 00:14:57.291885 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 00:14:58.199975 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260811 00:14:58.615634 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 00:14:58.903558 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:14:59.501315 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 00:14:59.898797 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 00:15:00.679190 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260811 00:15:01.566663 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 00:15:01.965991 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 00:15:02.276497 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 00:15:03.071630 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260811 00:15:03.952747 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:15:03.953950 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 00:15:04.261922 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 00:15:04.788081 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 00:15:05.840323 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260811 00:15:06.472509 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 00:15:06.490805 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 00:15:07.382223 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 00:15:08.451858 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260811 00:15:08.759642 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 00:15:08.974009 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:15:09.169978 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20260811 00:15:09.701059 19603 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:15:09.701238 19603 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:15:09.701880 19603 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:15:09.703255 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:15:09.703783 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:15:09.703925 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:15:09.704224 19603 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:15:10.114037 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 00:15:11.070775 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 00:15:11.274668 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260811 00:15:11.583644 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 00:15:12.826488 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 00:15:13.612774 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 00:15:13.909372 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260811 00:15:13.956544 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 00:15:14.191376 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:15:15.131986 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 00:15:16.159037 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:15:16.342645 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:15:16.371907 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260811 00:15:16.476197 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (60 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:15:16.512928 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 00:15:17.306457 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 00:15:18.586071 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 00:15:18.977056 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260811 00:15:19.124387 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 00:15:19.370817 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:15:19.922859 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 00:15:21.113206 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 00:15:21.289239 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260811 00:15:21.434077 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 00:15:22.525944 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 00:15:23.662768 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 00:15:23.663121 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260811 00:15:23.812832 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 00:15:24.719631 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:15:25.295095 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20260811 00:15:25.593719 19626 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:15:25.593961 19626 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:15:25.594664 19626 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:15:25.596297 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:15:25.596568 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:15:25.596678 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:15:25.596940 19626 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:15:26.159231 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260811 00:15:26.347745 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 00:15:26.372184 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 00:15:27.459072 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 00:15:28.660357 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260811 00:15:28.800356 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 00:15:28.971539 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 00:15:29.722627 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:15:29.723668 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 00:15:31.199219 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260811 00:15:31.217569 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 00:15:31.337668 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 00:15:32.279233 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 00:15:33.603709 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260811 00:15:33.628341 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 00:15:33.909447 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 00:15:34.844386 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:15:35.072074 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 00:15:35.847617 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260811 00:15:36.220345 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 00:15:36.559568 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 00:15:37.965911 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260811 00:15:38.516244 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20260811 00:15:38.595844 19626 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:15:38.596077 19626 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:15:38.596771 19626 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:15:38.598273 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:15:38.598738 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:15:38.598848 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:15:38.599059 19626 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:15:38.728287 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 00:15:38.918572 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 00:15:39.921420 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:15:40.414688 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 00:15:40.870380 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260811 00:15:41.033437 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 00:15:41.657435 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 00:15:43.071511 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 00:15:43.139595 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260811 00:15:43.654145 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 00:15:44.030820 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 00:15:45.014981 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:15:45.598898 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 00:15:45.929209 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260811 00:15:46.138931 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 00:15:46.585170 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 00:15:48.078299 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 00:15:48.280506 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260811 00:15:48.704650 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 00:15:49.160581 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 00:15:50.103385 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:15:50.336573 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 00:15:50.669484 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260811 00:15:51.396211 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 00:15:51.510200 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 00:15:52.763938 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 00:15:53.075909 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260811 00:15:53.957690 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 00:15:54.040338 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20260811 00:15:54.399296 19626 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:15:54.399570 19626 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:15:54.400250 19626 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:15:54.401891 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:15:54.402155 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:15:54.402267 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:15:54.402522 19626 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:15:55.111043 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:15:55.308307 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 00:15:55.430312 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260811 00:15:56.312418 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 00:15:56.578310 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 00:15:57.588788 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 00:15:57.900859 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260811 00:15:58.965934 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 00:15:59.316514 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 00:16:00.203390 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:16:00.204629 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 00:16:00.351857 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260811 00:16:01.349704 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20260811 00:16:01.436298 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:16:01.436568 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:16:01.437322 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:16:01.439095 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:16:01.439298 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:16:01.439446 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:16:01.439721 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:16:01.698810 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 00:16:02.631410 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 00:16:02.849231 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260811 00:16:03.976363 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 00:16:04.159200 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 00:16:04.929177 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 00:16:05.252640 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:16:05.253823 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260811 00:16:06.439272 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 00:16:06.735424 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 00:16:07.008052 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 00:16:07.816720 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260811 00:16:09.279886 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 00:16:09.555126 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 00:16:09.593449 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 00:16:10.279817 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:16:10.359218 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260811 00:16:11.667567 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 00:16:11.990518 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 00:16:12.006631 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 00:16:12.795085 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260811 00:16:14.166774 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 00:16:14.530519 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 00:16:14.659549 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 00:16:15.254624 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260811 00:16:15.615556 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:16:16.564487 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (120 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:16:16.829872 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20260811 00:16:16.961470 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:16:16.961750 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:16:16.962492 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:16:16.964082 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:16:16.964270 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:16:16.964406 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:16:16.964670 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:16:17.030829 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 00:16:17.345307 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:16:17.603622 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:16:17.641217 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260811 00:16:19.197016 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 00:16:19.222628 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 00:16:19.551126 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 00:16:20.101765 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260811 00:16:20.651949 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:16:21.469589 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 00:16:21.908236 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 00:16:22.018371 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 00:16:22.574440 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260811 00:16:24.261670 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 00:16:24.536767 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 00:16:24.707970 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 00:16:24.950677 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260811 00:16:25.809746 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:16:26.975221 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 00:16:27.028672 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 00:16:27.214609 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260811 00:16:27.634620 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20260811 00:16:29.228736 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:16:29.228979 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:16:29.229722 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:16:29.231557 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:16:29.231760 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:16:29.231865 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:16:29.232131 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:16:29.494110 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 00:16:29.731691 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 00:16:29.879184 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 00:16:30.106732 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260811 00:16:30.925511 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:16:31.956252 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 00:16:32.004303 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 00:16:32.332638 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 00:16:32.631422 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260811 00:16:34.446157 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 00:16:34.560595 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 00:16:34.695192 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 00:16:35.339246 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260811 00:16:35.986752 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:16:36.915273 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 00:16:37.095687 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 00:16:37.207542 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 00:16:37.924149 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260811 00:16:39.413089 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 00:16:39.574661 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 00:16:39.888240 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 00:16:40.140115 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260811 00:16:41.290719 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:16:41.819154 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 00:16:42.313920 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260811 00:16:42.558322 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 00:16:42.816133 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20260811 00:16:43.402951 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:16:43.403182 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:16:43.404027 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:16:43.406000 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:16:43.406203 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:16:43.406330 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:16:43.406565 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:16:44.609212 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 00:16:44.915108 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 00:16:45.138361 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 00:16:45.411180 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260811 00:16:46.412843 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:16:47.117291 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 00:16:47.156312 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 00:16:47.798985 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260811 00:16:47.980010 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 00:16:49.562898 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 00:16:49.809976 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 00:16:50.267374 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 00:16:50.444605 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260811 00:16:51.594815 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:16:52.106731 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 00:16:52.322157 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 00:16:52.715296 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 00:16:52.860541 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260811 00:16:54.753928 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 00:16:54.846369 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 00:16:55.038643 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 00:16:55.226559 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260811 00:16:56.633572 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:16:57.150661 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 00:16:57.270210 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 00:16:57.556192 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 00:16:57.740823 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260811 00:16:59.626489 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 00:16:59.871301 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 00:16:59.934181 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 00:17:00.194265 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260811 00:17:01.792117 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:17:02.098577 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 00:17:02.221995 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 00:17:02.400084 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20260811 00:17:02.635671 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:02.635967 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:02.636687 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:02.638620 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:17:02.638821 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:17:02.638926 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:02.639173 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:02.660059 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260811 00:17:04.433063 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 00:17:04.662134 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 00:17:05.137245 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 00:17:05.138960 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260811 00:17:06.817934 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:17:06.875707 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 00:17:07.235839 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 00:17:07.685711 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 00:17:07.979118 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260811 00:17:09.584779 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 00:17:09.792574 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 00:17:10.549466 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260811 00:17:10.568181 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 00:17:11.937305 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260811 00:17:11.960924 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 00:17:12.336303 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 00:17:12.941089 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260811 00:17:12.965018 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 00:17:14.491772 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 00:17:15.259130 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 00:17:15.458369 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260811 00:17:15.638757 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 00:17:16.649683 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (180 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:17:16.702211 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 00:17:17.101028 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:17:17.688822 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 00:17:17.991882 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260811 00:17:18.015280 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:17:18.908494 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:17:19.491897 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260811 00:17:19.944552 19636 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:19.944816 19636 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:19.945453 19636 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:19.947414 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:17:19.947629 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:17:19.947743 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:19.947955 19636 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:20.149195 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260811 00:17:20.273252 19636 consensus_queue.cc:579] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Leader has been unable to successfully communicate with peer b7cd95c75ac84415bbae31cced0aa1c0 for more than 300 seconds (300.257s)
W20260811 00:17:20.277796 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 00:17:20.329456 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260811 00:17:20.529735 19636 consensus_queue.cc:579] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Leader has been unable to successfully communicate with peer b7cd95c75ac84415bbae31cced0aa1c0 for more than 300 seconds (300.161s)
W20260811 00:17:22.067894 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 00:17:22.103693 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20260811 00:17:22.309010 19636 consensus_queue.cc:579] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Leader has been unable to successfully communicate with peer 2d01f4514a644a2ea81db4c9f506e19d for more than 300 seconds (300.415s)
W20260811 00:17:22.601341 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20260811 00:17:22.647735 19680 consensus_queue.cc:579] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de [LEADER]: Leader has been unable to successfully communicate with peer 2d01f4514a644a2ea81db4c9f506e19d for more than 300 seconds (300.493s)
W20260811 00:17:22.650054 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 00:17:22.852916 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260811 00:17:24.563831 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 00:17:25.024370 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 00:17:25.053511 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 00:17:25.467705 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260811 00:17:27.123795 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:17:27.271782 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 00:17:27.558482 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 00:17:27.650779 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20260811 00:17:27.725279 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:27.725477 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:27.726243 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:27.727928 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:17:27.728108 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:17:27.728220 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:27.728425 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:27.878482 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260811 00:17:29.890673 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 00:17:30.038821 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 00:17:30.249351 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 00:17:30.361274 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260811 00:17:32.193814 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:17:32.194823 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 00:17:32.410233 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 00:17:32.749534 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 00:17:33.156435 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260811 00:17:34.849531 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 00:17:34.996903 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 00:17:35.230841 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 00:17:35.788439 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260811 00:17:37.201844 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:17:37.416793 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 00:17:37.568795 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 00:17:37.713742 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 00:17:38.411415 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260811 00:17:39.920629 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 00:17:40.005321 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260811 00:17:40.140242 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 00:17:40.772008 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20260811 00:17:40.828840 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:40.829051 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:40.829816 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:40.831671 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:17:40.831813 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:17:40.831930 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:40.832165 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:42.373415 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:17:42.440559 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 00:17:42.469508 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260811 00:17:42.583506 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 00:17:43.156716 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20260811 00:17:44.196789 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:44.197077 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:44.197767 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:44.199493 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:17:44.199663 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:17:44.199781 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:44.200053 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:45.016324 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 00:17:45.262128 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 00:17:45.381363 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260811 00:17:45.408208 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 00:17:47.488345 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:17:47.604599 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 00:17:47.729972 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 00:17:47.950654 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 00:17:47.999889 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260811 00:17:50.271087 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 00:17:50.286968 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 00:17:50.319245 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260811 00:17:50.393781 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 00:17:52.567967 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:17:52.632550 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 00:17:52.673363 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 00:17:52.924849 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260811 00:17:53.045022 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 00:17:55.260591 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 00:17:55.380793 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20260811 00:17:55.535020 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:17:55.535281 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:17:55.536026 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:17:55.537777 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:17:55.537995 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:17:55.538115 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:17:55.538326 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:17:55.560427 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260811 00:17:55.631309 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 00:17:57.720709 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:17:57.753221 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 00:17:57.911599 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 00:17:58.172104 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 00:17:58.185516 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260811 00:18:00.138859 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 00:18:00.160416 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 00:18:00.761613 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260811 00:18:00.913432 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 00:18:02.425570 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 00:18:02.768986 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:18:02.770179 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 00:18:03.356896 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 00:18:03.470695 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260811 00:18:05.192930 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 00:18:05.206430 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 00:18:05.783205 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 00:18:06.190742 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260811 00:18:07.533416 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 00:18:07.737836 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 00:18:08.043915 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:18:08.285501 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 00:18:08.849926 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260811 00:18:10.085991 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 00:18:10.115289 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 00:18:10.822340 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 00:18:11.525736 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260811 00:18:12.448800 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 00:18:12.508988 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260811 00:18:13.057313 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:18:13.176105 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 00:18:13.670073 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20260811 00:18:14.077840 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:18:14.078219 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:18:14.078915 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:18:14.080822 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:18:14.081028 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:18:14.081151 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:18:14.081362 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:18:14.760726 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 00:18:14.832520 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260811 00:18:15.814126 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 00:18:16.051769 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20260811 00:18:16.269212 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:18:16.269508 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:18:16.270268 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:18:16.272019 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:18:16.272418 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:18:16.272526 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:18:16.272751 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:18:16.724550 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (242 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:18:17.246445 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 00:18:17.462096 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 00:18:18.082755 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:18:18.424175 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 00:18:18.441882 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260811 00:18:19.797031 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:18:20.226493 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:18:20.245867 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 00:18:20.987648 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260811 00:18:21.371862 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 00:18:21.950520 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 00:18:22.789798 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 00:18:23.226630 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:18:23.690727 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260811 00:18:24.185483 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 00:18:24.624648 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 00:18:25.066212 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20260811 00:18:25.403788 19679 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:18:25.404048 19679 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:18:25.404739 19679 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:18:25.406407 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:18:25.406778 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:18:25.406890 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:18:25.407208 19679 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:18:26.161537 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260811 00:18:26.649138 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 00:18:27.170027 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 00:18:27.522159 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 00:18:28.476671 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:18:28.633599 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260811 00:18:29.069485 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 00:18:29.514976 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 00:18:30.051744 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 00:18:30.904608 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260811 00:18:31.638839 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 00:18:32.192449 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 00:18:32.586947 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 00:18:33.569084 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:18:33.570209 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260811 00:18:34.112005 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 00:18:34.857493 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 00:18:35.145223 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 00:18:35.913504 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260811 00:18:36.982080 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 00:18:37.273741 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 00:18:37.726210 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 00:18:38.236701 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260811 00:18:38.673413 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:18:39.519783 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20260811 00:18:39.723465 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:18:39.723680 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:18:39.724436 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:18:39.725996 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:18:39.726280 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:18:39.726418 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:18:39.726677 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:18:39.758234 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 00:18:40.033413 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260811 00:18:40.847993 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20260811 00:18:42.060604 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:18:42.060886 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:18:42.061543 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:18:42.063109 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:18:42.063273 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:18:42.063395 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:18:42.063647 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:18:42.161859 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 00:18:42.300179 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 00:18:42.368949 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 00:18:43.253122 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260811 00:18:43.745424 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:18:44.687451 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 00:18:44.709960 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 00:18:44.868366 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 00:18:45.670390 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260811 00:18:46.976938 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 00:18:47.088205 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 00:18:47.559684 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 00:18:48.064399 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260811 00:18:48.940851 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:18:49.537551 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 00:18:49.668751 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 00:18:49.858552 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 00:18:50.346652 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260811 00:18:51.979955 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 00:18:52.384087 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 00:18:52.471848 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 00:18:52.799573 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260811 00:18:54.085686 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:18:54.677850 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 00:18:54.897512 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 00:18:55.154682 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 00:18:55.347841 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260811 00:18:57.257422 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 00:18:57.352063 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 00:18:57.759966 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 00:18:57.901518 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260811 00:18:59.354192 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:18:59.466773 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 00:19:00.010944 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 00:19:00.100445 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20260811 00:19:00.131085 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:00.131335 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:00.132164 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:00.133716 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:19:00.134017 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:19:00.134114 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:00.134315 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:00.385722 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260811 00:19:01.830693 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 00:19:02.512960 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 00:19:02.595556 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 00:19:03.131022 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260811 00:19:04.450080 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:19:04.451052 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 00:19:05.090838 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 00:19:05.312604 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 00:19:05.679137 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260811 00:19:07.086756 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 00:19:07.639822 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 00:19:08.004438 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 00:19:08.393352 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260811 00:19:09.581990 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:19:09.865314 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 00:19:10.069501 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 00:19:10.359050 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 00:19:10.730926 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260811 00:19:12.375756 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 00:19:12.510908 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 00:19:13.005978 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 00:19:13.293684 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260811 00:19:14.812824 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:19:14.822088 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 00:19:14.961421 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 00:19:15.258407 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260811 00:19:15.766275 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20260811 00:19:16.188689 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:16.188910 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:16.189626 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:16.191402 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:19:16.191573 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:19:16.191676 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:16.191874 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:16.809852 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (302 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:19:17.279968 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 00:19:17.551573 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 00:19:18.010053 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 00:19:18.167315 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260811 00:19:19.990641 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:19:19.991766 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 00:19:20.167354 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20260811 00:19:20.564133 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:20.564428 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:20.565178 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:20.566766 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:19:20.566905 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:19:20.566992 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:20.567245 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:20.599519 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260811 00:19:20.668023 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:19:21.499276 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:19:22.730481 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 00:19:22.787305 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 00:19:23.017041 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 00:19:23.166159 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260811 00:19:25.068001 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:19:25.154125 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 00:19:25.299081 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 00:19:25.480873 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 00:19:25.530897 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260811 00:19:27.609824 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 00:19:27.671233 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 00:19:27.882467 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 00:19:28.007995 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260811 00:19:30.038930 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 00:19:30.156440 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:19:30.157418 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 00:19:30.330403 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260811 00:19:30.416455 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20260811 00:19:31.935544 19708 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:31.935802 19708 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:31.936444 19708 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:31.938395 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:19:31.938555 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:19:31.938658 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:31.938856 19708 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:32.486096 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 00:19:32.547374 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 00:19:32.854704 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 00:19:32.998626 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260811 00:19:34.857168 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 00:19:35.096903 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 00:19:35.381546 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:19:35.474274 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 00:19:35.677524 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260811 00:19:37.558794 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 00:19:37.671464 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 00:19:38.260525 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 00:19:38.270886 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260811 00:19:40.035043 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 00:19:40.105822 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 00:19:40.466511 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:19:40.684839 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 00:19:40.847061 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260811 00:19:42.345448 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 00:19:42.682893 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 00:19:43.152581 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 00:19:43.188491 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260811 00:19:44.960419 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 00:19:45.205231 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 00:19:45.575271 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:19:45.576318 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20260811 00:19:45.649040 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:45.649245 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:45.649923 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:45.651618 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:19:45.651830 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:19:45.651937 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:45.652259 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:45.791101 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260811 00:19:47.387743 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 00:19:47.549674 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 00:19:48.022634 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 00:19:48.406193 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260811 00:19:49.872584 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 00:19:50.080547 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 00:19:50.858245 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:19:50.859306 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 00:19:51.034017 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260811 00:19:52.363368 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 00:19:52.498580 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 00:19:53.377969 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260811 00:19:53.777245 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20260811 00:19:54.102298 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:19:54.102522 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:19:54.103159 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:19:54.104868 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:19:54.105072 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:19:54.105175 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:19:54.105365 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:19:54.793965 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 00:19:55.023109 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 00:19:55.653071 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 00:19:55.871691 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:19:55.907943 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260811 00:19:57.420185 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 00:19:57.553694 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 00:19:58.176842 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 00:19:58.238768 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260811 00:20:00.029425 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 00:20:00.224056 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 00:20:00.615891 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 00:20:00.851543 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260811 00:20:01.024900 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:20:02.457319 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 00:20:02.520661 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 00:20:02.956233 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 00:20:03.387038 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260811 00:20:04.689472 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 00:20:05.034634 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 00:20:05.539659 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 00:20:05.857348 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260811 00:20:06.246531 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:20:06.937260 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 00:20:07.526106 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 00:20:07.783679 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 00:20:08.098110 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260811 00:20:09.474143 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 00:20:10.128775 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 00:20:10.348929 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260811 00:20:10.423414 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20260811 00:20:10.805591 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:20:10.805883 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:20:10.806670 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:20:10.808328 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:20:10.808514 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:20:10.808637 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:20:10.808897 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:20:11.285172 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:20:11.711031 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 00:20:12.833231 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 00:20:13.034562 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 00:20:13.351763 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260811 00:20:14.027319 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 00:20:15.397003 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 00:20:15.542191 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 00:20:16.067638 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260811 00:20:16.342127 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:20:16.520073 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 00:20:16.897773 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (362 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:20:17.931641 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 00:20:18.020409 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 00:20:18.649869 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260811 00:20:19.020951 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 00:20:20.461009 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 00:20:20.572306 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 00:20:21.129406 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260811 00:20:21.364596 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:20:21.564004 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:20:22.786088 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:20:22.890790 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 00:20:22.946862 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 00:20:23.412940 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260811 00:20:24.259819 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 00:20:25.310482 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 00:20:25.371762 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260811 00:20:25.989238 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20260811 00:20:26.354061 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:20:26.354338 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:20:26.355115 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:20:26.356849 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:20:26.356992 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:20:26.357095 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:20:26.357301 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:20:26.383647 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:20:26.744000 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 00:20:27.758695 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 00:20:28.011175 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 00:20:28.263679 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260811 00:20:29.275811 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 00:20:30.262167 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 00:20:30.638237 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 00:20:30.730350 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260811 00:20:31.477488 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:20:31.809969 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 00:20:32.840890 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 00:20:33.114108 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 00:20:33.324326 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260811 00:20:34.333035 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 00:20:35.038954 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 00:20:35.759685 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 00:20:35.803951 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260811 00:20:36.726814 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:20:36.728048 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 00:20:37.538347 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 00:20:38.245487 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 00:20:38.261586 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260811 00:20:39.410827 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20260811 00:20:39.474337 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:20:39.474614 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:20:39.475405 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:20:39.477128 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:20:39.477491 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:20:39.477689 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:20:39.477921 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:20:39.974179 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 00:20:40.438591 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 00:20:40.636518 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260811 00:20:41.748778 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:20:41.749799 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 00:20:42.487866 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 00:20:42.737874 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 00:20:43.114585 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260811 00:20:44.350755 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 00:20:45.110706 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 00:20:45.194880 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 00:20:45.607129 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260811 00:20:46.904604 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:20:46.905581 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 00:20:47.612982 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 00:20:47.699417 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 00:20:47.874222 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260811 00:20:49.432523 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 00:20:50.081243 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 00:20:50.086385 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 00:20:50.246136 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260811 00:20:51.985291 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:20:51.986372 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 00:20:52.729853 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 00:20:52.844589 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 00:20:52.846240 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260811 00:20:54.764146 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 00:20:55.131886 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 00:20:55.142771 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260811 00:20:55.552965 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20260811 00:20:55.843322 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:20:55.843618 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:20:55.844318 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:20:55.846156 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:20:55.846326 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:20:55.846487 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:20:55.846705 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:20:56.993952 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:20:57.001302 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 00:20:57.484078 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 00:20:57.647116 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 00:20:57.955188 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260811 00:20:59.529733 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 00:20:59.967026 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 00:21:00.548703 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 00:21:00.549294 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260811 00:21:02.084259 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260811 00:21:02.204569 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 00:21:02.627257 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 00:21:03.173533 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 00:21:03.178298 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260811 00:21:04.608543 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 00:21:04.986621 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 00:21:05.629060 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 00:21:05.683583 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260811 00:21:07.032819 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 00:21:07.234287 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:21:07.411535 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 00:21:08.143481 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 00:21:08.353384 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260811 00:21:09.474905 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 00:21:09.963145 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 00:21:10.893177 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 00:21:11.114238 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260811 00:21:11.745522 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 00:21:12.346335 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:21:12.347409 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20260811 00:21:13.144543 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:21:13.144742 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:21:13.145500 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:21:13.147383 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:21:13.147567 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:21:13.147691 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:21:13.147908 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:21:13.346398 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 00:21:13.469815 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260811 00:21:14.204586 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 00:21:14.881110 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 00:21:15.634548 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 00:21:15.737238 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260811 00:21:16.721692 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 00:21:16.983600 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (422 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:21:17.213276 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 00:21:17.609912 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:21:18.054445 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 00:21:18.395605 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260811 00:21:19.309270 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 00:21:19.729347 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 00:21:20.512933 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20260811 00:21:20.723420 19730 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:21:20.723668 19730 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:21:20.724407 19730 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:21:20.726001 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:21:20.726322 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:21:20.726447 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:21:20.726708 19730 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:21:21.180343 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260811 00:21:21.829341 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 00:21:22.100018 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 00:21:22.645052 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:21:22.891934 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260811 00:21:23.346490 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:21:24.082444 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:21:24.338617 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 00:21:24.672951 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 00:21:25.515169 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 00:21:26.027350 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260811 00:21:26.904816 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 00:21:27.024889 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 00:21:27.841729 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:21:27.889465 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 00:21:28.251925 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260811 00:21:29.385149 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 00:21:29.824030 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 00:21:29.964564 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 00:21:30.876045 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260811 00:21:32.076876 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 00:21:32.395593 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 00:21:32.610632 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 00:21:32.892423 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:21:33.449100 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260811 00:21:34.530848 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 00:21:34.852270 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 00:21:34.986773 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20260811 00:21:35.607142 19776 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:21:35.607365 19776 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:21:35.608002 19776 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:21:35.609704 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:21:35.609966 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:21:35.610076 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:21:35.610301 19776 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:21:35.964784 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260811 00:21:37.046794 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 00:21:37.235562 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 00:21:37.691581 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 00:21:37.913272 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:21:38.583765 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260811 00:21:39.387205 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 00:21:39.907033 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 00:21:40.341037 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 00:21:40.954453 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260811 00:21:41.858712 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 00:21:42.494623 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 00:21:42.950708 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:21:42.970288 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 00:21:43.502915 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260811 00:21:44.366143 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 00:21:44.723636 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 00:21:45.501796 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 00:21:45.930670 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260811 00:21:46.863144 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 00:21:47.251121 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 00:21:48.098831 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:21:48.099852 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260811 00:21:48.217826 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20260811 00:21:48.917372 19780 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:21:48.917587 19780 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:21:48.918265 19780 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:21:48.920074 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:21:48.920311 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:21:48.920414 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:21:48.920620 19780 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:21:49.468864 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 00:21:49.717542 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 00:21:50.573405 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 00:21:50.711489 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260811 00:21:51.900810 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 00:21:52.507575 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 00:21:52.895864 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 00:21:53.081231 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260811 00:21:53.349866 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:21:54.330016 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 00:21:55.086689 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 00:21:55.261884 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 00:21:55.383594 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260811 00:21:56.956102 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 00:21:57.760190 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 00:21:57.783846 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 00:21:57.981070 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260811 00:21:58.507161 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:21:59.383788 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 00:22:00.085057 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 00:22:00.125797 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 00:22:00.590725 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260811 00:22:01.778652 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 00:22:02.418057 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 00:22:02.777004 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 00:22:02.809368 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260811 00:22:03.671509 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:22:04.216711 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 00:22:04.735980 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 00:22:05.195618 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260811 00:22:05.230607 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20260811 00:22:06.039036 19780 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:22:06.039292 19780 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:22:06.040210 19780 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:22:06.042213 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:22:06.042433 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:22:06.042533 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:22:06.042806 19780 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:22:06.637833 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 00:22:07.606307 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 00:22:07.636276 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 00:22:07.895620 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260811 00:22:08.782842 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:22:08.889839 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 00:22:09.791631 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 00:22:10.258821 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 00:22:10.488723 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260811 00:22:11.574888 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 00:22:12.274750 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 00:22:12.530656 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 00:22:13.055765 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260811 00:22:13.815629 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:22:13.816812 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 00:22:15.042629 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 00:22:15.165377 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 00:22:15.260743 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260811 00:22:16.418833 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 00:22:17.069464 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (482 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
I20260811 00:22:17.082592 19784 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:22:17.082959 19784 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:22:17.083750 19784 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:22:17.085542 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:22:17.085862 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:22:17.085965 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:22:17.086297 19784 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:22:17.604470 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 00:22:17.759208 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 00:22:17.810590 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260811 00:22:18.905536 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:22:19.044348 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 00:22:19.937948 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 00:22:20.265178 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 00:22:20.356189 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260811 00:22:21.302053 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 00:22:22.602016 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 00:22:22.735874 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 00:22:22.805344 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260811 00:22:23.948727 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:22:23.949858 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 00:22:24.978322 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 00:22:25.182646 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:22:25.379899 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:22:25.491999 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260811 00:22:26.341005 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 00:22:27.523275 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 00:22:27.664271 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 00:22:27.808732 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260811 00:22:28.638332 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 00:22:28.954447 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:22:29.931577 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 00:22:30.174738 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 00:22:30.299706 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260811 00:22:31.237008 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
I20260811 00:22:31.287683 19784 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:22:31.287923 19784 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:22:31.288576 19784 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:22:31.290527 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:22:31.290725 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:22:31.290868 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:22:31.291107 19784 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:22:32.207261 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 00:22:32.621256 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 00:22:32.880892 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260811 00:22:33.687307 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 00:22:34.142676 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:22:34.548069 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 00:22:35.232244 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 00:22:35.248871 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260811 00:22:36.135871 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
I20260811 00:22:36.337935 19784 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:22:36.338197 19784 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:22:36.338979 19784 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:22:36.340636 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:22:36.340823 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:22:36.340945 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:22:36.341251 19784 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:22:37.013944 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 00:22:37.554235 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 00:22:37.644886 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260811 00:22:38.791934 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 00:22:39.288853 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:22:39.612712 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 00:22:40.047609 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 00:22:40.400408 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260811 00:22:41.466867 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 00:22:42.394155 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 00:22:42.617959 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 00:22:42.914932 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260811 00:22:44.148643 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 00:22:44.436622 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:22:45.016028 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 00:22:45.219725 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 00:22:45.553551 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260811 00:22:46.661581 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 00:22:47.525454 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 00:22:47.812899 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 00:22:48.067124 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260811 00:22:49.040581 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 00:22:49.502452 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:22:50.038300 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 00:22:50.308794 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 00:22:50.400432 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260811 00:22:51.408560 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 00:22:52.298540 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 00:22:52.835510 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 00:22:52.873035 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260811 00:22:53.748813 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 00:22:54.611968 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:22:54.997926 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 00:22:55.115434 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260811 00:22:55.585978 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
I20260811 00:22:56.037587 19784 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:22:56.037845 19784 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:22:56.038548 19784 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:22:56.040239 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:22:56.040416 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:22:56.040529 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:22:56.040814 19784 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:22:56.161661 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 00:22:57.287633 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 00:22:57.401644 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 00:22:58.321349 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260811 00:22:58.499838 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 00:22:59.785002 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:22:59.786051 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 00:23:00.160845 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 00:23:00.808799 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260811 00:23:01.054638 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 00:23:02.426265 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 00:23:02.626431 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 00:23:03.515221 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260811 00:23:03.667047 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 00:23:04.785390 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:04.788071 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 00:23:05.085496 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260811 00:23:05.796969 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
I20260811 00:23:06.130652 19804 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:23:06.130939 19804 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:23:06.131626 19804 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:23:06.133275 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:23:06.133491 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:23:06.133647 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:23:06.133929 19804 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:23:06.206380 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 00:23:07.242061 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 00:23:07.705102 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 00:23:08.112237 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260811 00:23:08.846789 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 00:23:09.788707 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:23:09.790006 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 00:23:10.215811 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 00:23:10.510089 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260811 00:23:11.119273 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 00:23:12.552627 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 00:23:12.656761 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 00:23:13.010576 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260811 00:23:13.599155 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 00:23:14.869786 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:14.870836 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 00:23:15.133751 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 00:23:15.774010 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260811 00:23:16.316071 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 00:23:17.042335 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 00:23:17.155339 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (542 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:23:17.627912 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 00:23:18.242486 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260811 00:23:18.921361 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 00:23:19.606071 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 00:23:19.942323 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:19.943399 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
I20260811 00:23:21.030695 19804 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:23:21.030905 19804 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:23:21.031639 19804 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:23:21.033391 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:23:21.033563 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:23:21.033699 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:23:21.033941 19804 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:23:21.167794 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260811 00:23:21.473364 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 00:23:22.015250 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 00:23:22.407760 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 00:23:23.526468 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260811 00:23:24.021123 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 00:23:24.573262 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 00:23:24.911051 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 00:23:25.052698 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:23:26.158900 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260811 00:23:26.515465 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:23:26.722199 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:23:26.856339 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 00:23:27.525882 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
I20260811 00:23:28.455899 19804 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:23:28.456111 19804 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:23:28.456736 19804 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:23:28.458560 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:23:28.458784 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:23:28.458906 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:23:28.459172 19804 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:23:28.697788 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260811 00:23:28.864671 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 00:23:29.064374 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 00:23:30.059486 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:23:30.060678 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 00:23:31.255997 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260811 00:23:31.527452 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 00:23:31.687626 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 00:23:32.573066 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 00:23:33.593858 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260811 00:23:33.826915 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 00:23:34.453833 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 00:23:35.060072 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:35.061220 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 00:23:36.126221 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260811 00:23:36.311595 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 00:23:36.952741 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 00:23:37.589232 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 00:23:38.709298 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260811 00:23:38.725580 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 00:23:39.586645 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 00:23:40.177088 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:40.285319 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 00:23:41.119524 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 00:23:41.306638 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260811 00:23:42.068535 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 00:23:42.881877 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
I20260811 00:23:43.211213 19828 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:23:43.211439 19828 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:23:43.212146 19828 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:23:43.213989 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:23:43.214146 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:23:43.214272 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:23:43.214558 19828 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:23:43.512878 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 00:23:43.994134 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260811 00:23:44.491118 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 00:23:45.289533 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:23:45.384073 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 00:23:46.250419 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 00:23:46.528551 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260811 00:23:46.864604 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 00:23:47.705693 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 00:23:48.666091 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 00:23:48.937214 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260811 00:23:49.501278 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 00:23:50.366343 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:50.444427 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 00:23:51.183815 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 00:23:51.444509 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260811 00:23:52.121984 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 00:23:52.981848 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 00:23:53.931679 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 00:23:54.002921 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260811 00:23:54.674754 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 00:23:55.376638 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:23:55.377791 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 00:23:56.528483 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 00:23:56.591805 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260811 00:23:57.091184 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 00:23:57.539528 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 00:23:59.211050 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 00:23:59.308295 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260811 00:23:59.589777 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 00:23:59.925278 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 00:24:00.405784 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260811 00:24:00.445048 19828 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:00.445286 19828 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:00.446060 19828 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:00.447916 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:24:00.448231 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:24:00.448380 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:00.460983 19828 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:01.503185 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 00:24:01.955050 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260811 00:24:02.180027 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 00:24:02.509279 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 00:24:03.834335 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 00:24:04.450402 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260811 00:24:04.650621 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 00:24:04.955484 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 00:24:05.561141 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:24:06.567062 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 00:24:07.197005 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260811 00:24:07.218915 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 00:24:07.505483 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 00:24:08.972504 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 00:24:09.664788 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 00:24:09.925269 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 00:24:09.994697 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260811 00:24:10.700214 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:24:11.267700 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 00:24:12.369153 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 00:24:12.511385 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 00:24:12.761929 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260811 00:24:13.723256 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 00:24:14.888756 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
I20260811 00:24:15.081810 19834 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:15.082079 19834 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:15.082772 19834 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:15.084571 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:24:15.084834 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:24:15.084959 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:15.085220 19834 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:15.094630 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 00:24:15.454250 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260811 00:24:15.818009 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:24:16.360790 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 00:24:17.240924 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (602 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:24:17.327003 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 00:24:17.661846 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 00:24:18.128742 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260811 00:24:19.077046 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
I20260811 00:24:19.538353 19834 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:19.538580 19834 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:19.539398 19834 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:19.541005 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:24:19.541172 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:24:19.541318 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:19.541595 19834 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:19.861579 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 00:24:20.081343 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 00:24:20.646878 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260811 00:24:20.848577 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:24:21.631265 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 00:24:22.450734 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 00:24:22.580088 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 00:24:23.095660 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260811 00:24:23.972560 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 00:24:25.095703 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 00:24:25.154317 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260811 00:24:25.512953 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260811 00:24:25.860721 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:24:26.308972 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 00:24:27.218555 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 00:24:27.478446 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 00:24:27.961457 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:24:28.023981 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:24:28.798285 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 00:24:29.878861 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 00:24:30.114940 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 00:24:30.565233 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260811 00:24:30.895684 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:24:31.390975 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 00:24:32.484285 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 00:24:32.512346 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 00:24:33.105490 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260811 00:24:33.694975 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 00:24:35.010547 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 00:24:35.069694 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 00:24:35.751997 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260811 00:24:36.016840 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:24:36.017987 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 00:24:37.544806 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
I20260811 00:24:37.562801 19834 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:37.563026 19834 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:37.563735 19834 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:37.565547 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:24:37.565773 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:24:37.565943 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:37.566231 19834 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:37.609221 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 00:24:38.000159 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260811 00:24:38.296555 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 00:24:40.113595 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 00:24:40.236136 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 00:24:40.583748 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 00:24:40.625429 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260811 00:24:41.092998 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:24:42.656932 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 00:24:42.969406 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 00:24:43.179616 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260811 00:24:43.326634 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 00:24:44.699661 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 00:24:45.470754 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260811 00:24:45.516755 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
I20260811 00:24:45.540588 19855 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:45.540846 19855 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:45.541523 19855 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:45.543110 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:24:45.543279 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:24:45.543397 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:45.543658 19855 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:46.114248 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:24:46.115510 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 00:24:46.862391 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 00:24:47.907295 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260811 00:24:47.948113 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 00:24:48.893848 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 00:24:49.274719 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 00:24:50.296437 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260811 00:24:50.341754 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 00:24:51.261487 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:24:51.542588 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
I20260811 00:24:52.016355 19857 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:52.016677 19857 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:52.017431 19857 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:52.019266 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:24:52.019495 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:24:52.019618 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:52.019863 19857 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:52.111009 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 00:24:52.490151 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260811 00:24:52.622371 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 00:24:54.147969 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 00:24:54.595952 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 00:24:54.932835 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260811 00:24:55.014986 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
I20260811 00:24:55.608724 19857 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:24:55.608953 19857 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:24:55.609714 19857 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:24:55.611291 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:24:55.611507 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:24:55.611651 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:24:55.611905 19857 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:24:56.283385 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:24:56.982244 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 00:24:57.071249 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 00:24:57.224859 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260811 00:24:57.338629 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 00:24:59.496646 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 00:24:59.591629 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260811 00:24:59.701979 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 00:24:59.985785 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 00:25:01.403636 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:25:01.855918 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 00:25:02.010013 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260811 00:25:02.049477 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 00:25:02.649205 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 00:25:04.218664 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 00:25:04.296625 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 00:25:04.612134 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260811 00:25:04.863842 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 00:25:06.474354 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:25:06.677345 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 00:25:06.765033 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 00:25:07.206382 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260811 00:25:07.693965 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 00:25:08.978850 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 00:25:09.241876 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 00:25:09.879629 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260811 00:25:10.117993 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
I20260811 00:25:10.996284 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:10.996510 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:10.997236 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:10.998801 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:25:10.999053 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:25:10.999161 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:10.999426 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:11.359043 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 00:25:11.707028 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:25:11.807045 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 00:25:12.258833 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260811 00:25:12.503371 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 00:25:14.147012 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 00:25:14.173251 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 00:25:14.858206 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260811 00:25:15.227938 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260811 00:25:16.620733 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 00:25:16.817612 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:25:16.818718 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 00:25:17.328390 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (662 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:25:17.481796 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 00:25:17.690351 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
I20260811 00:25:18.414171 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:18.414443 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:18.415165 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:18.417033 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:25:18.417186 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:25:18.417289 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:18.417516 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:19.076651 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 00:25:19.535956 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 00:25:19.730623 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260811 00:25:20.013433 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260811 00:25:21.444900 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 00:25:21.801820 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 00:25:22.064893 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:25:22.091378 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 00:25:22.703984 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
I20260811 00:25:24.003381 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:24.003647 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:24.004355 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:24.006132 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:25:24.006333 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:25:24.006494 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:24.006806 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:24.151151 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 00:25:24.327741 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 00:25:24.487668 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 00:25:24.982720 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260811 00:25:26.780224 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 00:25:26.824807 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 00:25:26.858767 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 00:25:27.243923 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:25:27.398948 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260811 00:25:29.018051 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 00:25:29.263247 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 00:25:29.302664 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:25:29.316468 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:25:29.722787 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260811 00:25:31.473598 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 00:25:31.835299 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 00:25:32.039572 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 00:25:32.263171 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:25:32.264129 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260811 00:25:34.162873 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 00:25:34.243606 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 00:25:34.674124 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260811 00:25:34.865356 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260811 00:25:36.797495 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 00:25:36.813874 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 00:25:36.955826 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 00:25:37.278913 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260811 00:25:37.544754 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
I20260811 00:25:37.669543 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:37.669919 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:37.670621 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:37.672281 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:25:37.672453 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:25:37.672607 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:37.672888 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:39.135035 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 00:25:39.318076 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 00:25:39.349478 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 00:25:40.003455 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260811 00:25:41.617735 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 00:25:41.759872 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 00:25:41.761382 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
I20260811 00:25:42.353734 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:42.354022 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:42.354719 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:42.355527 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260811 00:25:42.356563 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:25:42.356698 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:25:42.356835 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:42.357110 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:42.607137 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260811 00:25:44.063828 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 00:25:44.152670 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 00:25:44.560073 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 00:25:45.054355 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260811 00:25:46.322881 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 00:25:46.599776 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 00:25:47.064648 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 00:25:47.363445 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:25:47.865494 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260811 00:25:48.988097 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 00:25:49.296916 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 00:25:49.569943 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 00:25:50.365372 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260811 00:25:51.469594 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 00:25:51.898177 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 00:25:52.060197 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 00:25:52.422582 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:25:52.766426 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260811 00:25:54.046955 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 00:25:54.315542 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 00:25:54.573848 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260811 00:25:55.338924 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260811 00:25:56.657817 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 00:25:56.941784 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 00:25:57.066663 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 00:25:57.465492 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:25:57.850978 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
I20260811 00:25:58.314165 19864 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:25:58.314363 19864 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:25:58.315112 19864 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:25:58.316828 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:25:58.317060 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:25:58.317201 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:25:58.317482 19864 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:25:59.069422 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 00:25:59.630345 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 00:25:59.825344 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 00:26:00.262015 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260811 00:26:01.347268 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 00:26:02.240288 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 00:26:02.343875 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 00:26:02.592621 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:26:02.705241 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260811 00:26:03.494859 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 00:26:04.594465 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 00:26:04.845243 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 00:26:04.928215 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260811 00:26:06.198925 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 00:26:07.055048 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 00:26:07.318733 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 00:26:07.561419 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260811 00:26:07.655788 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:26:08.961462 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 00:26:09.797808 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 00:26:09.842288 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 00:26:10.141819 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260811 00:26:11.255522 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 00:26:12.202922 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 00:26:12.229883 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260811 00:26:12.509166 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260811 00:26:12.700661 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:26:13.721431 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 00:26:14.592375 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 00:26:14.717533 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 00:26:15.088117 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
I20260811 00:26:15.657896 19892 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:26:15.658114 19892 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:26:15.658818 19892 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:26:15.660449 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:26:15.660645 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:26:15.660759 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:26:15.661039 19892 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:26:16.197961 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 00:26:17.042935 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 00:26:17.238793 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 00:26:17.414880 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (722 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
I20260811 00:26:17.430282 19892 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:26:17.430605 19892 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:26:17.431325 19892 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:26:17.432941 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:26:17.433185 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:26:17.433353 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:26:17.433673 19892 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:26:17.460114 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260811 00:26:17.838634 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:26:19.045490 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 00:26:19.791610 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 00:26:19.893867 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 00:26:20.076838 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260811 00:26:21.426573 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 00:26:22.064330 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 00:26:22.321851 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 00:26:22.713099 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260811 00:26:22.987347 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:26:23.880195 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 00:26:24.831806 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 00:26:24.897401 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 00:26:25.213548 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260811 00:26:26.192710 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 00:26:27.409240 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 00:26:27.770808 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260811 00:26:28.006819 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260811 00:26:28.007809 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260811 00:26:28.698734 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 00:26:29.547660 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 00:26:30.024986 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 00:26:30.486148 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+-------------
<unknown> (127.15.174.126:46603) | 127.15.174.126:46603 | UNAVAILABLE
Error from 127.15.174.126:46603: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
FAILED
I20260811 00:26:30.584522 16057 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.15.174.126:46603, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:26:31.094543 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 00:26:32.212384 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 00:26:32.582093 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 00:26:32.877271 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260811 00:26:33.069597 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:26:33.721263 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 00:26:34.595978 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 00:26:34.827965 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
I20260811 00:26:35.282392 19892 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:26:35.282666 19892 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:26:35.283342 19892 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:26:35.285116 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:26:35.285277 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:26:35.285449 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:26:35.285780 19892 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:26:35.635586 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260811 00:26:36.321141 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 00:26:36.856050 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 00:26:37.360100 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 00:26:38.158325 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:26:38.159353 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260811 00:26:38.831949 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 00:26:39.357749 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 00:26:39.795495 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 00:26:40.907101 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260811 00:26:41.297524 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 00:26:42.101756 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 00:26:42.144285 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 00:26:43.237018 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260811 00:26:43.656548 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260811 00:26:43.958133 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 00:26:44.485173 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 00:26:44.897257 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
************************ BEGIN STACKS **************************
[New LWP 16060]
[New LWP 16061]
[New LWP 16062]
[New LWP 16590]
[New LWP 16591]
[New LWP 16592]
[New LWP 16593]
[New LWP 19899]
[New LWP 19900]
[New LWP 19901]
[New LWP 19903]
[New LWP 19906]
[New LWP 19907]
[New LWP 19908]
[New LWP 19909]
[New LWP 19911]
I20260811 00:26:45.696415 19892 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:26:45.696652 19892 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:26:45.697429 19892 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:26:45.699200 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:26:45.699337 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:26:45.699440 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:26:45.699678 19892 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:26:46.365763 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260811 00:26:46.577133 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 00:26:46.905678 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 00:26:47.247944 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 00:26:48.448164 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:26:48.623247 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260811 00:26:48.969578 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 00:26:49.513442 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 00:26:49.557374 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 00:26:51.149680 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260811 00:26:51.491158 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 00:26:52.087598 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 00:26:52.267738 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 00:26:53.681869 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260811 00:26:53.683101 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260811 00:26:53.691980 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 00:26:54.699266 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 00:26:55.122493 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 00:26:56.146497 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260811 00:26:56.217401 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 00:26:57.410756 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 00:26:57.434930 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 00:26:58.620944 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 00:26:58.638149 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260811 00:26:58.995963 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260811 00:26:59.665073 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 00:27:00.220039 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 00:27:01.164541 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 00:27:01.212252 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260811 00:27:02.047925 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 00:27:02.926506 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 00:27:03.653002 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 00:27:03.717597 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260811 00:27:04.090709 16466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20260811 00:27:04.208166 19892 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:27:04.208427 19892 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:27:04.209199 19892 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:27:04.211129 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
W20260811 00:27:04.211944 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
I20260811 00:27:04.212071 16466 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:27:04.212337 19892 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:27:04.673092 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 00:27:05.726321 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 00:27:05.957228 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 00:27:06.123018 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260811 00:27:07.144178 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 00:27:08.150120 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 00:27:08.601468 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260811 00:27:08.626600 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260811 00:27:09.253839 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260811 00:27:09.779850 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 00:27:10.557780 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 00:27:10.947379 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 00:27:11.001037 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260811 00:27:12.060567 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 00:27:13.144912 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260811 00:27:13.369534 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260811 00:27:13.509506 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260811 00:27:14.428928 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260811 00:27:14.433444 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
0x00007f42913f9ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f42898fad40 (LWP 16057) "all_types-itest" 0x00007f42913f9ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f42854bd700 (LWP 16060) "all_types-itest" 0x00007f42913fe32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f4283ef4700 (LWP 16061) "kernel-watcher-" 0x00007f42913f9fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f42836f3700 (LWP 16062) "rpc reactor-160" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f4282ef2700 (LWP 16590) "rpc reactor-165" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f42824ea700 (LWP 16591) "rpc reactor-165" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f4281ce9700 (LWP 16592) "rpc reactor-165" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f42814e8700 (LWP 16593) "rpc reactor-165" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f427e451700 (LWP 19899) "rpc reactor-198" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f427fb31700 (LWP 19900) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f427ab5c700 (LWP 19901) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f427cbc8700 (LWP 19903) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f427eea4700 (LWP 19906) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f427db48700 (LWP 19907) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f427c3b7700 (LWP 19908) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f427bbb6700 (LWP 19909) "rpc reactor-199" 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f427b3b5700 (LWP 19911) "client-negotiat" 0x00007f42913f9fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f427b3b5700 (LWP 19911)):
#0 0x00007f42913f9fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f428e5e48d9 in kudu::ConditionVariable::WaitFor (this=0x7f42913f9fb9 <pthread_cond_timedwait@@GLIBC_2.3.2+649>, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007f428e894e0a in kudu::ThreadPool::DispatchThread (this=0x616000011480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f428e89891c in kudu::ThreadPool::CreateThread()::$_2::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
W20260811 00:27:15.595616 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#4 0x00007f428e898799 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_2>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
W20260811 00:27:15.736340 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#5 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x61100090a028) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#6 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f427bbb6700 (LWP 19909)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619001713d80, timeout=0.099383809138089418) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619001713d80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x6140003eea88, flags=7197696, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x6140003eea80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cc728) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f427c3b7700 (LWP 19908)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619001713880, timeout=0.099363892339169979) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619001713880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x61400033b088, flags=7196800, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x61400033b080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cc5e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f427db48700 (LWP 19907)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619001713380, timeout=0.099344560177996755) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619001713380, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x61400033ae88, flags=7195904, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x61400033ae80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cc4a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f427eea4700 (LWP 19906)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619001712e80, timeout=0.09956520120613277) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619001712e80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x6140003e7088, flags=7195008, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x6140003e7080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cc368) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f427cbc8700 (LWP 19903)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000062c80, timeout=0.099977214122191072) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000062c80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x6140003f8e88, flags=7192320, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x6140003f8e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cb6e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f427ab5c700 (LWP 19901)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000063180, timeout=0.099012063117697835) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000063180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x6140003f8c88, flags=7191424, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x6140003f8c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cb5a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f427fb31700 (LWP 19900)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000063680, timeout=0.098998738219961524) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000063680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000338088, flags=7190528, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000338080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cb468) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f427e451700 (LWP 19899)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000063b80, timeout=0.099064785288646817) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000063b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000337e88, flags=7189632, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000337e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110008cb328) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f42814e8700 (LWP 16593)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000055a80, timeout=0.099007038166746497) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000055a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000004288, flags=16256, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110000589e8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f4281ce9700 (LWP 16592)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000055580, timeout=0.099472834728658199) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000055580, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000004088, flags=15360, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000004080) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x6110000588a8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260811 00:27:16.215356 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f42824ea700 (LWP 16591)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000055080, timeout=0.098905102582648396) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000055080, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000003e88, flags=14464, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003e80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x611000058768) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f4282ef2700 (LWP 16590)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000054b80, timeout=0.099966572597622871) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000054b80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000003c88, flags=13568, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003c80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x611000058628) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f42836f3700 (LWP 16062)):
#0 0x00007f428a6cd947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f428c7bcbda in epoll_poll (loop=0x619000052880, timeout=0.099985981825739145) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f428c7c0228 in ev_run (loop=0x619000052880, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f4294e58e31 in ev::loop_ref::run (this=this@entry=0x614000003488, flags=9088, flags@entry=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f4294e51326 in kudu::rpc::ReactorThread::RunThread (this=0x614000003480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f4294e563ac in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f4294e56229 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x611000051fa8) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f4283ef4700 (LWP 16061)):
#0 0x00007f42913f9fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f428e5e450e in kudu::ConditionVariable::WaitUntil (this=0x612000009118, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
#2 0x00007f42a243b02c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f42a24334aa in kudu::CountDownLatch::WaitFor (this=0x612000009140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f428e71a45f in kudu::KernelStackWatchdog::RunThread (this=0x612000009040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f428e71b51c in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f428e71b399 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:297
#7 0x00007f42a2df3bbd in std::function<void ()>::operator()() const (this=0x611000052228) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:687
#8 0x00007f428e873eaf in kudu::Thread::SuperviseThread (arg=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
I20260811 00:27:16.659827 19934 raft_consensus.cc:482] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d01f4514a644a2ea81db4c9f506e19d)
I20260811 00:27:16.660138 19934 raft_consensus.cc:504] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7cd95c75ac84415bbae31cced0aa1c0" member_type: VOTER last_known_addr { host: "127.15.174.65" port: 37601 } } peers { permanent_uuid: "2d01f4514a644a2ea81db4c9f506e19d" member_type: VOTER last_known_addr { host: "127.15.174.66" port: 46197 } } peers { permanent_uuid: "29d3790ed6ae4e509b332ca347d6f5de" member_type: VOTER last_known_addr { host: "127.15.174.67" port: 45711 } }
I20260811 00:27:16.661203 19934 leader_election.cc:288] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601), 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197)
W20260811 00:27:16.663051 16466 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111)
W20260811 00:27:16.663358 16465 leader_election.cc:334] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111)
I20260811 00:27:16.663702 16465 leader_election.cc:302] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29d3790ed6ae4e509b332ca347d6f5de; no voters: 2d01f4514a644a2ea81db4c9f506e19d, b7cd95c75ac84415bbae31cced0aa1c0
I20260811 00:27:16.664126 19934 raft_consensus.cc:2726] T 8de88d670149435a8031520dfd947788 P 29d3790ed6ae4e509b332ca347d6f5de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 00:27:16.857095 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
Thread 2 (Thread 0x7f42854bd700 (LWP 16060)):
#0 0x00007f42913fe32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000039b4fb in __interceptor_waitpid (pid=<optimized out>, status=0x7f42854bb8b0, options=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f428e85e796 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#3 0x00007f428e85e4c5 in kudu::Subprocess::Wait (this=0x7f42854bb8b0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#4 0x00007f428e6f42be in kudu::PstackWatcher::RunStackDump (argv=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#5 0x00007f428e6f39a7 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f428e6f2e96 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f428e6f1417 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000005386f9 in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#9 0x0000000000538583 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#10 0x0000000000538503 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#11 0x00000000005384d9 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#12 0x0000000000538499 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000029c18) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#13 0x000000000053839b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#14 0x00007f4295d3f8cf in execute_native_thread_routine () from /tmp/dist-test-taskdqxoSS/build/asan/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#15 0x00007f42913f36db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f428a6cd61f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f42898fad40 (LWP 16057)):
#0 0x00007f42913f9ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f428e5e41e7 in kudu::ConditionVariable::Wait (this=0x60e000144010) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
#2 0x00007f42a34aa1b9 in kudu::CountDownLatch::Wait (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260811 00:27:17.509130 16586 heartbeater.cc:644] Failed to heartbeat to 127.15.174.126:46603 (782 consecutive failures): Network error: Failed to ping master at 127.15.174.126:46603: Client connection negotiation failed: client connection to 127.15.174.126:46603: connect: Connection refused (error 111)
W20260811 00:27:17.879909 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#3 0x00007f42a2ba0f4b in kudu::Synchronizer::Wait (this=0x7ffdb914d3e0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f429be41ef3 in kudu::client::KuduClient::Data::ConnectToCluster (this=0x615000c2fb80, client=<optimized out>, deadline=..., creds_policy=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260811 00:27:18.352097 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260811 00:27:18.511987 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
#5 0x00007f429bdc1901 in kudu::client::KuduClientBuilder::Build (this=0x7ffdb914dce0, client=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
#6 0x00007f429c767eac in kudu::tools::CreateKuduClient (master_addresses=..., client=<optimized out>, can_see_all_replicas=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
W20260811 00:27:19.343528 16465 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260811 00:27:19.430485 16465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111) [suppressed 42 similar messages]
#7 0x00007f42a2460c15 in kudu::tools::RemoteKsckCluster::Connect (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:528
#8 0x00007f42a23d487e in kudu::tools::Ksck::CheckClusterRunning (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f42a23d683f in kudu::tools::Ksck::Run (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f42a23db13d in kudu::tools::Ksck::RunAndPrintResults (this=0x619001712990) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
W20260811 00:27:20.429417 16466 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#11 0x00007f42a3452dd3 in kudu::ClusterVerifier::RunKsck (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f42a3452556 in kudu::ClusterVerifier::CheckCluster (this=0x7ffdb914f100) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x0000000000472628 in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x60c00000d000, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
#14 0x000000000050e27d in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)5> > >::TestBody (this=<optimized out>) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:804
W20260811 00:27:20.766413 16466 consensus_peers.cc:487] T f4dfa5ed7be8439a88f32ac097f9c56d P 29d3790ed6ae4e509b332ca347d6f5de -> Peer 2d01f4514a644a2ea81db4c9f506e19d (127.15.174.66:46197): Couldn't send request to peer 2d01f4514a644a2ea81db4c9f506e19d. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.66:46197: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
W20260811 00:27:21.180685 16465 consensus_peers.cc:487] T 1c04612d65bb419893570a7a89c44b60 P 29d3790ed6ae4e509b332ca347d6f5de -> Peer b7cd95c75ac84415bbae31cced0aa1c0 (127.15.174.65:37601): Couldn't send request to peer b7cd95c75ac84415bbae31cced0aa1c0. Status: Network error: Client connection negotiation failed: client connection to 127.15.174.65:37601: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
#15 0x00007f428f0cdbfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f428f0d3a28 "the test body", method=<optimized out>, object=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x60c00000d000, method=<optimized out>, location=location@entry=0x7f428f0d3a28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f428f0c25db in testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x60c00000d000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f428f0c2795 in testing::TestInfo::Run (this=0x6110000471c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f428f0c2cdf in testing::TestSuite::Run (this=0x611000045a00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x611000045a00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f428f0c329f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f428f0ce13e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f428f0d55a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x61600000b480) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x61600000b480, method=<optimized out>, location=location@entry=0x7f428f0d55a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f428f0c283d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x0000000000539482 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x0000000000537992 in main (argc=<optimized out>, argv=0x7ffdb914f988) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20260811 00:27:21.335047 16060 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786408041 (unix time) try "date -d @1786408041" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003eb9) received by PID 16057 (TID 0x7f42854bd700) from PID 16057; stack trace: ***
@ 0x7f428d2bca46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f42913fe980 (unknown) at ??:0
@ 0x7f428a5eae87 gsignal at ??:0
@ 0x7f428a5ec7f1 abort at ??:0
@ 0x7f428d2ae9ad google::LogMessage::Fail() at ??:0
@ 0x7f428d2b188a google::LogMessage::SendToLog() at ??:0
@ 0x7f428d2ae4de google::LogMessage::Flush() at ??:0
@ 0x7f428d2afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x538891 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63
@ 0x538583 std::__invoke_impl<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
@ 0x538503 std::__invoke<>() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
@ 0x5384d9 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
@ 0x538499 std::thread::_Invoker<>::operator()() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
@ 0x53839b std::thread::_State_impl<>::_M_run() at /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
@ 0x7f4295d3f8cf execute_native_thread_routine at ??:0
@ 0x7f42913f36db start_thread at ??:0
@ 0x7f428a6cd61f clone at ??:0