Note: This is test shard 2 of 8.
[==========] Running 3 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AllTypesItest/0, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)1> >
[ RUN ] AllTypesItest/0.TestTimestampPadding
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 09:41:20.945802 32048 test_util.cc:268] Using random seed: 1674436184
2026-08-12T09:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:41:20Z Disabled control of system clock
I20260812 09:41:20.972828 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.76.62:42737
--webserver_interface=127.31.76.62
--webserver_port=0
--builtin_ntp_servers=127.31.76.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.76.62:42737 with env {}
W20260812 09:41:21.127540 32058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:21.127843 32058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:21.127878 32058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:21.132055 32058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:41:21.132120 32058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:21.132138 32058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:41:21.132154 32058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:41:21.137387 32058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.76.62:42737
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.76.62:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.76.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:21.138983 32058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:21.141153 32058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:21.155278 32065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:21.155537 32058 server_base.cc:1034] running on GCE node
W20260812 09:41:21.156695 32064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:21.156970 32067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:21.157828 32058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:21.159637 32058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:21.160861 32058 hybrid_clock.cc:648] HybridClock initialized: now 1786527681160824 us; error 79 us; skew 500 ppm
I20260812 09:41:21.163606 32058 webserver.cc:460] Webserver started at http://127.31.76.62:46759/ using document root <none> and password file <none>
I20260812 09:41:21.164525 32058 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:21.164597 32058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:21.164945 32058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:21.167698 32058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/instance:
uuid: "3358de19bef549789c89bed8d1dc277c"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.168324 32058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal/instance:
uuid: "3358de19bef549789c89bed8d1dc277c"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.173557 32058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260812 09:41:21.177176 32077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.178866 32058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 09:41:21.179086 32058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
uuid: "3358de19bef549789c89bed8d1dc277c"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.179211 32058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:21.199293 32058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:21.200592 32058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:21.200927 32058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:21.218467 32058 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.62:42737
I20260812 09:41:21.218538 32150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.62:42737 every 8 connection(s)
I20260812 09:41:21.220011 32058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
I20260812 09:41:21.222362 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 32058
I20260812 09:41:21.222615 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal/instance
I20260812 09:41:21.226538 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:21.238510 32151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c: Bootstrap starting.
I20260812 09:41:21.244534 32151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:21.246695 32151 log.cc:824] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:21.250296 32151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c: No bootstrap required, opened a new log
I20260812 09:41:21.257625 32151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } }
I20260812 09:41:21.257903 32151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:21.257967 32151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3358de19bef549789c89bed8d1dc277c, State: Initialized, Role: FOLLOWER
I20260812 09:41:21.258589 32151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } }
I20260812 09:41:21.258760 32151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:41:21.258845 32151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:41:21.258948 32151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:21.260083 32151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } }
I20260812 09:41:21.260623 32151 leader_election.cc:302] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3358de19bef549789c89bed8d1dc277c; no voters:
I20260812 09:41:21.261046 32151 leader_election.cc:288] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:41:21.261322 32159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:21.261876 32159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [term 1 LEADER]: Becoming Leader. State: Replica: 3358de19bef549789c89bed8d1dc277c, State: Running, Role: LEADER
I20260812 09:41:21.262334 32159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } }
I20260812 09:41:21.263067 32151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:41:21.265515 32160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3358de19bef549789c89bed8d1dc277c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } } }
I20260812 09:41:21.265705 32160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [sys.catalog]: This master's current role is: LEADER
I20260812 09:41:21.266191 32163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3358de19bef549789c89bed8d1dc277c. Latest consensus state: current_term: 1 leader_uuid: "3358de19bef549789c89bed8d1dc277c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3358de19bef549789c89bed8d1dc277c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 42737 } } }
I20260812 09:41:21.266290 32163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c [sys.catalog]: This master's current role is: LEADER
I20260812 09:41:21.267246 32174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:41:21.272181 32174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:41:21.280205 32174 catalog_manager.cc:1347] Generated new cluster ID: 408cc606559c44fa89264763b25effa1
I20260812 09:41:21.280303 32174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:41:21.311170 32174 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:41:21.312201 32174 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:41:21.321779 32174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3358de19bef549789c89bed8d1dc277c: Generated new TSK 0
I20260812 09:41:21.322834 32174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:41:21.336134 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.1:0
--local_ip_for_outbound_sockets=127.31.76.1
--webserver_interface=127.31.76.1
--webserver_port=0
--tserver_master_addrs=127.31.76.62:42737
--builtin_ntp_servers=127.31.76.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260812 09:41:21.513749 32187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:21.514058 32187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:21.514103 32187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:21.518507 32187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:21.518585 32187 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:21.518602 32187 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:21.518680 32187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.1
I20260812 09:41:21.523725 32187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.76.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:42737
--never_fsync=true
--heap_profile_path=/tmp/kudu.32187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.1
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:21.525028 32187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:21.526721 32187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:21.539791 32196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:21.539897 32194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:21.539810 32192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:21.540378 32187 server_base.cc:1034] running on GCE node
I20260812 09:41:21.540844 32187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:21.541644 32187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:21.542832 32187 hybrid_clock.cc:648] HybridClock initialized: now 1786527681542814 us; error 28 us; skew 500 ppm
I20260812 09:41:21.545811 32187 webserver.cc:460] Webserver started at http://127.31.76.1:37761/ using document root <none> and password file <none>
I20260812 09:41:21.546567 32187 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:21.546644 32187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:21.546921 32187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:21.549309 32187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/instance:
uuid: "5938211537904981950a78f870e72c73"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.550117 32187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal/instance:
uuid: "5938211537904981950a78f870e72c73"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.554949 32187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 09:41:21.558030 32203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.559578 32187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 09:41:21.559736 32187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
uuid: "5938211537904981950a78f870e72c73"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.559859 32187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:21.583740 32187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:21.585041 32187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:21.585361 32187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:21.586329 32187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:21.589834 32187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:21.589923 32187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.590039 32187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:21.590101 32187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.615237 32187 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.1:35055
I20260812 09:41:21.615330 32372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.1:35055 every 8 connection(s)
I20260812 09:41:21.616482 32187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
I20260812 09:41:21.617615 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 32187
I20260812 09:41:21.617730 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal/instance
I20260812 09:41:21.620596 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.2:0
--local_ip_for_outbound_sockets=127.31.76.2
--webserver_interface=127.31.76.2
--webserver_port=0
--tserver_master_addrs=127.31.76.62:42737
--builtin_ntp_servers=127.31.76.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 09:41:21.632656 32373 heartbeater.cc:347] Connected to a master server at 127.31.76.62:42737
I20260812 09:41:21.632949 32373 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:21.633569 32373 heartbeater.cc:511] Master 127.31.76.62:42737 requested a full tablet report, sending...
I20260812 09:41:21.636279 32101 ts_manager.cc:194] Registered new tserver with Master: 5938211537904981950a78f870e72c73 (127.31.76.1:35055)
I20260812 09:41:21.638973 32101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.1:39045
W20260812 09:41:21.804890 32376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:21.805184 32376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:21.805226 32376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:21.809391 32376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:21.809465 32376 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:21.809481 32376 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:21.809517 32376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.2
I20260812 09:41:21.815018 32376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.76.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:42737
--never_fsync=true
--heap_profile_path=/tmp/kudu.32376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.2
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:21.816136 32376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:21.817677 32376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:21.829195 32387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:21.829233 32385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:21.829231 32381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:21.829540 32376 server_base.cc:1034] running on GCE node
I20260812 09:41:21.830153 32376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:21.830830 32376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:21.832026 32376 hybrid_clock.cc:648] HybridClock initialized: now 1786527681831992 us; error 47 us; skew 500 ppm
I20260812 09:41:21.834424 32376 webserver.cc:460] Webserver started at http://127.31.76.2:40509/ using document root <none> and password file <none>
I20260812 09:41:21.835040 32376 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:21.835144 32376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:21.835391 32376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:21.844913 32376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/instance:
uuid: "aab0c8793d474af3b206bb0c9c8aa0dd"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.845674 32376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal/instance:
uuid: "aab0c8793d474af3b206bb0c9c8aa0dd"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.850045 32376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260812 09:41:21.853186 32396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.854650 32376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:41:21.854805 32376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
uuid: "aab0c8793d474af3b206bb0c9c8aa0dd"
format_stamp: "Formatted at 2026-08-12 09:41:21 on dist-test-slave-bw88"
I20260812 09:41:21.854924 32376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:21.877315 32376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:21.878470 32376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:21.878703 32376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:21.879587 32376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:21.880707 32376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:21.880780 32376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.880856 32376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:21.880899 32376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:21.905011 32376 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.2:40977
I20260812 09:41:21.905210 32563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.2:40977 every 8 connection(s)
I20260812 09:41:21.906318 32376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
I20260812 09:41:21.913618 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 32376
I20260812 09:41:21.913740 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal/instance
I20260812 09:41:21.916133 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.3:0
--local_ip_for_outbound_sockets=127.31.76.3
--webserver_interface=127.31.76.3
--webserver_port=0
--tserver_master_addrs=127.31.76.62:42737
--builtin_ntp_servers=127.31.76.20:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 09:41:21.920065 32564 heartbeater.cc:347] Connected to a master server at 127.31.76.62:42737
I20260812 09:41:21.920357 32564 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:21.920925 32564 heartbeater.cc:511] Master 127.31.76.62:42737 requested a full tablet report, sending...
I20260812 09:41:21.922153 32101 ts_manager.cc:194] Registered new tserver with Master: aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2:40977)
I20260812 09:41:21.922837 32101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.2:47887
W20260812 09:41:22.091125 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:22.091457 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:22.091570 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:22.095844 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:22.095958 32568 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:22.096020 32568 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:22.096093 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.3
I20260812 09:41:22.101576 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.76.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:42737
--never_fsync=true
--heap_profile_path=/tmp/kudu.32568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.3
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:22.102738 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:22.104288 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:22.116263 32575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:22.116426 32576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:22.116487 32579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:22.116958 32568 server_base.cc:1034] running on GCE node
I20260812 09:41:22.117448 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:22.118142 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:22.119323 32568 hybrid_clock.cc:648] HybridClock initialized: now 1786527682119301 us; error 32 us; skew 500 ppm
I20260812 09:41:22.121803 32568 webserver.cc:460] Webserver started at http://127.31.76.3:37439/ using document root <none> and password file <none>
I20260812 09:41:22.122380 32568 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:22.122444 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:22.122622 32568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:22.124681 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/instance:
uuid: "60e461db0aa1482cb01ca3c8bb095cd8"
format_stamp: "Formatted at 2026-08-12 09:41:22 on dist-test-slave-bw88"
I20260812 09:41:22.125289 32568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal/instance:
uuid: "60e461db0aa1482cb01ca3c8bb095cd8"
format_stamp: "Formatted at 2026-08-12 09:41:22 on dist-test-slave-bw88"
I20260812 09:41:22.129510 32568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 09:41:22.132227 32586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:22.133577 32568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 09:41:22.133703 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
uuid: "60e461db0aa1482cb01ca3c8bb095cd8"
format_stamp: "Formatted at 2026-08-12 09:41:22 on dist-test-slave-bw88"
I20260812 09:41:22.133812 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:22.147511 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:22.148427 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:22.148641 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:22.149533 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:22.150661 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:22.150712 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:22.150781 32568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:22.150817 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:22.174261 32568 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.3:34681
I20260812 09:41:22.174383 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.3:34681 every 8 connection(s)
I20260812 09:41:22.175452 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
I20260812 09:41:22.177284 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 32568
I20260812 09:41:22.177431 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal/instance
I20260812 09:41:22.188024 32743 heartbeater.cc:347] Connected to a master server at 127.31.76.62:42737
I20260812 09:41:22.188304 32743 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:22.188859 32743 heartbeater.cc:511] Master 127.31.76.62:42737 requested a full tablet report, sending...
I20260812 09:41:22.189977 32101 ts_manager.cc:194] Registered new tserver with Master: 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3:34681)
I20260812 09:41:22.190558 32101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.3:35729
I20260812 09:41:22.192355 32048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:41:22.209618 32101 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44512:
name: "all-types-table"
schema {
columns {
name: "key"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000*\004\001\000\000\000T"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:41:22.213620 32101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:41:22.239924 32470 tablet_service.cc:1467] Processing CreateTablet for tablet ce91946b5af8449ba2ec0160bf809d5a (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION VALUES < 42
I20260812 09:41:22.239342 32271 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8080e52199402ebd951934e7e04ce2 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260812 09:41:22.240972 32470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce91946b5af8449ba2ec0160bf809d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.241932 32469 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8080e52199402ebd951934e7e04ce2 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260812 09:41:22.242331 32469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8080e52199402ebd951934e7e04ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.242743 32655 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8080e52199402ebd951934e7e04ce2 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 42 <= VALUES < 84
I20260812 09:41:22.243377 32654 tablet_service.cc:1467] Processing CreateTablet for tablet 1113e6c1419b4c58bfbd52945ef120f7 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260812 09:41:22.239426 32468 tablet_service.cc:1467] Processing CreateTablet for tablet 1113e6c1419b4c58bfbd52945ef120f7 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260812 09:41:22.243697 32655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8080e52199402ebd951934e7e04ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.243855 32270 tablet_service.cc:1467] Processing CreateTablet for tablet 1113e6c1419b4c58bfbd52945ef120f7 (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION 84 <= VALUES
I20260812 09:41:22.244122 32654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1113e6c1419b4c58bfbd52945ef120f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.244680 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1113e6c1419b4c58bfbd52945ef120f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.243976 32468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1113e6c1419b4c58bfbd52945ef120f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.250216 32272 tablet_service.cc:1467] Processing CreateTablet for tablet ce91946b5af8449ba2ec0160bf809d5a (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION VALUES < 42
I20260812 09:41:22.250615 32272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce91946b5af8449ba2ec0160bf809d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.252702 303 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Bootstrap starting.
I20260812 09:41:22.254554 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8080e52199402ebd951934e7e04ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.239455 32656 tablet_service.cc:1467] Processing CreateTablet for tablet ce91946b5af8449ba2ec0160bf809d5a (DEFAULT_TABLE table=all-types-table [id=64fd161ce8c74af6ad619f9fe5e2c311]), partition=RANGE (key) PARTITION VALUES < 42
I20260812 09:41:22.255923 32656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce91946b5af8449ba2ec0160bf809d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:22.256943 303 tablet_bootstrap.cc:654] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.258814 303 log.cc:824] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:22.260228 306 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: Bootstrap starting.
I20260812 09:41:22.261364 303 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: No bootstrap required, opened a new log
I20260812 09:41:22.261583 303 ts_tablet_manager.cc:1397] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260812 09:41:22.261773 304 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Bootstrap starting.
I20260812 09:41:22.264415 306 tablet_bootstrap.cc:654] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.266279 304 tablet_bootstrap.cc:654] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.267035 306 log.cc:824] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:22.267263 303 raft_consensus.cc:348] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.267588 303 raft_consensus.cc:374] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.267661 303 raft_consensus.cc:729] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab0c8793d474af3b206bb0c9c8aa0dd, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.268038 304 log.cc:824] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:22.268165 303 consensus_queue.cc:260] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.269467 303 ts_tablet_manager.cc:1428] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 09:41:22.269843 306 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: No bootstrap required, opened a new log
I20260812 09:41:22.269984 303 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd: Bootstrap starting.
I20260812 09:41:22.270043 306 ts_tablet_manager.cc:1397] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260812 09:41:22.270030 32564 heartbeater.cc:503] Master 127.31.76.62:42737 was elected leader, sending a full tablet report...
I20260812 09:41:22.270409 304 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: No bootstrap required, opened a new log
I20260812 09:41:22.270586 304 ts_tablet_manager.cc:1397] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260812 09:41:22.271911 303 tablet_bootstrap.cc:654] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.274212 303 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd: No bootstrap required, opened a new log
I20260812 09:41:22.274322 303 ts_tablet_manager.cc:1397] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 09:41:22.275206 303 raft_consensus.cc:348] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.275386 303 raft_consensus.cc:374] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.275432 303 raft_consensus.cc:729] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab0c8793d474af3b206bb0c9c8aa0dd, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.275552 303 consensus_queue.cc:260] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.275874 303 ts_tablet_manager.cc:1428] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260812 09:41:22.276144 303 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd: Bootstrap starting.
I20260812 09:41:22.277056 304 raft_consensus.cc:348] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.277359 304 raft_consensus.cc:374] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.277422 304 raft_consensus.cc:729] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e461db0aa1482cb01ca3c8bb095cd8, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.277626 306 raft_consensus.cc:348] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.277943 306 raft_consensus.cc:374] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.277958 304 consensus_queue.cc:260] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.278081 306 raft_consensus.cc:729] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938211537904981950a78f870e72c73, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.278678 303 tablet_bootstrap.cc:654] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.278652 306 consensus_queue.cc:260] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.279134 32743 heartbeater.cc:503] Master 127.31.76.62:42737 was elected leader, sending a full tablet report...
I20260812 09:41:22.279170 304 ts_tablet_manager.cc:1428] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20260812 09:41:22.279491 304 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8: Bootstrap starting.
I20260812 09:41:22.280979 32373 heartbeater.cc:503] Master 127.31.76.62:42737 was elected leader, sending a full tablet report...
I20260812 09:41:22.281103 306 ts_tablet_manager.cc:1428] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260812 09:41:22.281390 304 tablet_bootstrap.cc:654] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.281554 306 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73: Bootstrap starting.
I20260812 09:41:22.283452 306 tablet_bootstrap.cc:654] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.283748 304 tablet_bootstrap.cc:492] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8: No bootstrap required, opened a new log
I20260812 09:41:22.283859 304 ts_tablet_manager.cc:1397] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260812 09:41:22.284773 304 raft_consensus.cc:348] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.284987 304 raft_consensus.cc:374] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.285037 304 raft_consensus.cc:729] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e461db0aa1482cb01ca3c8bb095cd8, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.285159 304 consensus_queue.cc:260] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.285513 304 ts_tablet_manager.cc:1428] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 09:41:22.285717 304 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8: Bootstrap starting.
I20260812 09:41:22.286119 306 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73: No bootstrap required, opened a new log
I20260812 09:41:22.286268 306 ts_tablet_manager.cc:1397] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260812 09:41:22.286392 303 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd: No bootstrap required, opened a new log
I20260812 09:41:22.286567 303 ts_tablet_manager.cc:1397] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260812 09:41:22.287514 306 raft_consensus.cc:348] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.287575 303 raft_consensus.cc:348] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.287724 303 raft_consensus.cc:374] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.287771 303 raft_consensus.cc:729] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab0c8793d474af3b206bb0c9c8aa0dd, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.287747 306 raft_consensus.cc:374] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.287925 306 raft_consensus.cc:729] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938211537904981950a78f870e72c73, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.287902 303 consensus_queue.cc:260] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.288044 304 tablet_bootstrap.cc:654] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.288230 303 ts_tablet_manager.cc:1428] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 09:41:22.288237 306 consensus_queue.cc:260] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.288693 306 ts_tablet_manager.cc:1428] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 09:41:22.288924 306 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: Bootstrap starting.
I20260812 09:41:22.290266 304 tablet_bootstrap.cc:492] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8: No bootstrap required, opened a new log
I20260812 09:41:22.290383 304 ts_tablet_manager.cc:1397] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 09:41:22.291047 304 raft_consensus.cc:348] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.291169 306 tablet_bootstrap.cc:654] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:22.291164 304 raft_consensus.cc:374] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.291210 304 raft_consensus.cc:729] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e461db0aa1482cb01ca3c8bb095cd8, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.291374 304 consensus_queue.cc:260] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.291846 304 ts_tablet_manager.cc:1428] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260812 09:41:22.294039 306 tablet_bootstrap.cc:492] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: No bootstrap required, opened a new log
I20260812 09:41:22.294142 306 ts_tablet_manager.cc:1397] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 09:41:22.294859 306 raft_consensus.cc:348] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.294991 306 raft_consensus.cc:374] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:22.295017 306 raft_consensus.cc:729] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938211537904981950a78f870e72c73, State: Initialized, Role: FOLLOWER
I20260812 09:41:22.295113 306 consensus_queue.cc:260] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.295373 306 ts_tablet_manager.cc:1428] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:41:22.360875 317 raft_consensus.cc:482] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:22.361164 317 raft_consensus.cc:504] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.362761 317 leader_election.cc:288] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2:40977), 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3:34681)
W20260812 09:41:22.368599 32374 tablet.cc:2369] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:41:22.369841 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8080e52199402ebd951934e7e04ce2" candidate_uuid: "5938211537904981950a78f870e72c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" is_pre_election: true
I20260812 09:41:22.370054 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8080e52199402ebd951934e7e04ce2" candidate_uuid: "5938211537904981950a78f870e72c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" is_pre_election: true
I20260812 09:41:22.370303 32681 raft_consensus.cc:2454] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938211537904981950a78f870e72c73 in term 0.
I20260812 09:41:22.370523 32502 raft_consensus.cc:2454] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5938211537904981950a78f870e72c73 in term 0.
I20260812 09:41:22.371026 32206 leader_election.cc:302] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938211537904981950a78f870e72c73, 60e461db0aa1482cb01ca3c8bb095cd8; no voters:
I20260812 09:41:22.371421 317 raft_consensus.cc:2781] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:22.371573 317 raft_consensus.cc:482] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:22.371680 317 raft_consensus.cc:3037] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.373107 317 raft_consensus.cc:504] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.373633 317 leader_election.cc:288] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [CANDIDATE]: Term 1 election: Requested vote from peers aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2:40977), 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3:34681)
I20260812 09:41:22.374130 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8080e52199402ebd951934e7e04ce2" candidate_uuid: "5938211537904981950a78f870e72c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd"
I20260812 09:41:22.374190 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8080e52199402ebd951934e7e04ce2" candidate_uuid: "5938211537904981950a78f870e72c73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e461db0aa1482cb01ca3c8bb095cd8"
I20260812 09:41:22.374313 32502 raft_consensus.cc:3037] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.374472 32681 raft_consensus.cc:3037] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.375926 32502 raft_consensus.cc:2454] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938211537904981950a78f870e72c73 in term 1.
I20260812 09:41:22.376211 32681 raft_consensus.cc:2454] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5938211537904981950a78f870e72c73 in term 1.
I20260812 09:41:22.376286 32204 leader_election.cc:302] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938211537904981950a78f870e72c73, aab0c8793d474af3b206bb0c9c8aa0dd; no voters:
I20260812 09:41:22.376564 317 raft_consensus.cc:2781] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:22.376838 317 raft_consensus.cc:686] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [term 1 LEADER]: Becoming Leader. State: Replica: 5938211537904981950a78f870e72c73, State: Running, Role: LEADER
I20260812 09:41:22.377215 317 consensus_queue.cc:237] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.381174 32100 catalog_manager.cc:5495] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5938211537904981950a78f870e72c73 (127.31.76.1). New cstate: current_term: 1 leader_uuid: "5938211537904981950a78f870e72c73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } health_report { overall_health: UNKNOWN } } }
W20260812 09:41:22.408123 32565 tablet.cc:2369] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:41:22.426853 32745 tablet.cc:2369] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:41:22.444707 313 raft_consensus.cc:482] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:22.444903 313 raft_consensus.cc:504] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.446380 313 leader_election.cc:288] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5938211537904981950a78f870e72c73 (127.31.76.1:35055), 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3:34681)
I20260812 09:41:22.453135 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1113e6c1419b4c58bfbd52945ef120f7" candidate_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" is_pre_election: true
I20260812 09:41:22.453327 32681 raft_consensus.cc:2454] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab0c8793d474af3b206bb0c9c8aa0dd in term 0.
I20260812 09:41:22.453615 32300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1113e6c1419b4c58bfbd52945ef120f7" candidate_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5938211537904981950a78f870e72c73" is_pre_election: true
I20260812 09:41:22.453899 32300 raft_consensus.cc:2454] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aab0c8793d474af3b206bb0c9c8aa0dd in term 0.
I20260812 09:41:22.453930 32405 leader_election.cc:302] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e461db0aa1482cb01ca3c8bb095cd8, aab0c8793d474af3b206bb0c9c8aa0dd; no voters:
I20260812 09:41:22.454264 313 raft_consensus.cc:2781] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:22.454362 313 raft_consensus.cc:482] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:22.454404 313 raft_consensus.cc:3037] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.455783 313 raft_consensus.cc:504] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.456214 313 leader_election.cc:288] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [CANDIDATE]: Term 1 election: Requested vote from peers 5938211537904981950a78f870e72c73 (127.31.76.1:35055), 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3:34681)
I20260812 09:41:22.456615 32300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1113e6c1419b4c58bfbd52945ef120f7" candidate_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5938211537904981950a78f870e72c73"
I20260812 09:41:22.456725 32300 raft_consensus.cc:3037] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.456676 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1113e6c1419b4c58bfbd52945ef120f7" candidate_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e461db0aa1482cb01ca3c8bb095cd8"
I20260812 09:41:22.456840 32681 raft_consensus.cc:3037] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.457978 32681 raft_consensus.cc:2454] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab0c8793d474af3b206bb0c9c8aa0dd in term 1.
I20260812 09:41:22.458297 32300 raft_consensus.cc:2454] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aab0c8793d474af3b206bb0c9c8aa0dd in term 1.
I20260812 09:41:22.458369 32405 leader_election.cc:302] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e461db0aa1482cb01ca3c8bb095cd8, aab0c8793d474af3b206bb0c9c8aa0dd; no voters:
I20260812 09:41:22.458580 313 raft_consensus.cc:2781] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:22.458904 313 raft_consensus.cc:686] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 LEADER]: Becoming Leader. State: Replica: aab0c8793d474af3b206bb0c9c8aa0dd, State: Running, Role: LEADER
I20260812 09:41:22.459276 313 consensus_queue.cc:237] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } }
I20260812 09:41:22.463073 32100 catalog_manager.cc:5495] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd reported cstate change: term changed from 0 to 1, leader changed from <none> to aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2). New cstate: current_term: 1 leader_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } health_report { overall_health: UNKNOWN } } }
I20260812 09:41:22.476351 315 raft_consensus.cc:482] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:22.476508 315 raft_consensus.cc:504] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.477883 315 leader_election.cc:288] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2:40977), 5938211537904981950a78f870e72c73 (127.31.76.1:35055)
I20260812 09:41:22.482649 32300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce91946b5af8449ba2ec0160bf809d5a" candidate_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5938211537904981950a78f870e72c73" is_pre_election: true
I20260812 09:41:22.482869 32300 raft_consensus.cc:2454] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e461db0aa1482cb01ca3c8bb095cd8 in term 0.
I20260812 09:41:22.483091 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce91946b5af8449ba2ec0160bf809d5a" candidate_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" is_pre_election: true
I20260812 09:41:22.483287 32502 raft_consensus.cc:2454] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60e461db0aa1482cb01ca3c8bb095cd8 in term 0.
I20260812 09:41:22.483362 32592 leader_election.cc:302] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5938211537904981950a78f870e72c73, 60e461db0aa1482cb01ca3c8bb095cd8; no voters:
I20260812 09:41:22.483724 315 raft_consensus.cc:2781] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:22.483807 315 raft_consensus.cc:482] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:22.483879 315 raft_consensus.cc:3037] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.485528 315 raft_consensus.cc:504] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.485903 315 leader_election.cc:288] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [CANDIDATE]: Term 1 election: Requested vote from peers aab0c8793d474af3b206bb0c9c8aa0dd (127.31.76.2:40977), 5938211537904981950a78f870e72c73 (127.31.76.1:35055)
I20260812 09:41:22.486443 32300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce91946b5af8449ba2ec0160bf809d5a" candidate_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5938211537904981950a78f870e72c73"
I20260812 09:41:22.486559 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce91946b5af8449ba2ec0160bf809d5a" candidate_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd"
I20260812 09:41:22.486683 32300 raft_consensus.cc:3037] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.486682 32502 raft_consensus.cc:3037] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:22.488049 32502 raft_consensus.cc:2454] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e461db0aa1482cb01ca3c8bb095cd8 in term 1.
I20260812 09:41:22.488046 32300 raft_consensus.cc:2454] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60e461db0aa1482cb01ca3c8bb095cd8 in term 1.
I20260812 09:41:22.488387 32587 leader_election.cc:302] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60e461db0aa1482cb01ca3c8bb095cd8, aab0c8793d474af3b206bb0c9c8aa0dd; no voters:
I20260812 09:41:22.488584 315 raft_consensus.cc:2781] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:22.488962 315 raft_consensus.cc:686] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 LEADER]: Becoming Leader. State: Replica: 60e461db0aa1482cb01ca3c8bb095cd8, State: Running, Role: LEADER
I20260812 09:41:22.489439 315 consensus_queue.cc:237] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } }
I20260812 09:41:22.492980 32100 catalog_manager.cc:5495] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60e461db0aa1482cb01ca3c8bb095cd8 (127.31.76.3). New cstate: current_term: 1 leader_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 } health_report { overall_health: UNKNOWN } } }
I20260812 09:41:22.568277 32300 raft_consensus.cc:1261] T ce91946b5af8449ba2ec0160bf809d5a P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Refusing update from remote peer 60e461db0aa1482cb01ca3c8bb095cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.568342 32502 raft_consensus.cc:1261] T ce91946b5af8449ba2ec0160bf809d5a P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Refusing update from remote peer 60e461db0aa1482cb01ca3c8bb095cd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.569038 32300 raft_consensus.cc:1261] T 1113e6c1419b4c58bfbd52945ef120f7 P 5938211537904981950a78f870e72c73 [term 1 FOLLOWER]: Refusing update from remote peer aab0c8793d474af3b206bb0c9c8aa0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.569478 342 consensus_queue.cc:1035] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.570165 342 consensus_queue.cc:1035] T ce91946b5af8449ba2ec0160bf809d5a P 60e461db0aa1482cb01ca3c8bb095cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.570443 32681 raft_consensus.cc:1261] T 1113e6c1419b4c58bfbd52945ef120f7 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Refusing update from remote peer aab0c8793d474af3b206bb0c9c8aa0dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.572947 337 consensus_queue.cc:1035] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5938211537904981950a78f870e72c73" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.573655 337 consensus_queue.cc:1035] T 1113e6c1419b4c58bfbd52945ef120f7 P aab0c8793d474af3b206bb0c9c8aa0dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.574421 32681 raft_consensus.cc:1261] T 1a8080e52199402ebd951934e7e04ce2 P 60e461db0aa1482cb01ca3c8bb095cd8 [term 1 FOLLOWER]: Refusing update from remote peer 5938211537904981950a78f870e72c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.576296 351 consensus_queue.cc:1035] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e461db0aa1482cb01ca3c8bb095cd8" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.576886 348 mvcc.cc:205] Tried to move back new op lower bound from 7317617387786801152 to 7317617387480031232. Current Snapshot: MvccSnapshot[applied={T|T < 7317617387786801152}]
I20260812 09:41:22.577848 32502 raft_consensus.cc:1261] T 1a8080e52199402ebd951934e7e04ce2 P aab0c8793d474af3b206bb0c9c8aa0dd [term 1 FOLLOWER]: Refusing update from remote peer 5938211537904981950a78f870e72c73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 09:41:22.578415 351 consensus_queue.cc:1035] T 1a8080e52199402ebd951934e7e04ce2 P 5938211537904981950a78f870e72c73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab0c8793d474af3b206bb0c9c8aa0dd" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:22.586145 366 mvcc.cc:205] Tried to move back new op lower bound from 7317617387788333056 to 7317617387356372992. Current Snapshot: MvccSnapshot[applied={T|T < 7317617387788333056}]
I20260812 09:41:22.588513 350 mvcc.cc:205] Tried to move back new op lower bound from 7317617387806924800 to 7317617387020292096. Current Snapshot: MvccSnapshot[applied={T|T < 7317617387806924800}]
I20260812 09:41:22.676910 32656 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 09:41:22.679029 32470 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 09:41:22.679386 32270 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
3358de19bef549789c89bed8d1dc277c | 127.31.76.62:42737 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:33665 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5938211537904981950a78f870e72c73 | 127.31.76.1:35055 | HEALTHY | <none> | 1 | 0
60e461db0aa1482cb01ca3c8bb095cd8 | 127.31.76.3:34681 | HEALTHY | <none> | 1 | 0
aab0c8793d474af3b206bb0c9c8aa0dd | 127.31.76.2:40977 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 1 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.31.76.1 | experimental | 127.31.76.1:35055
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:40977
local_ip_for_outbound_sockets | 127.31.76.3 | experimental | 127.31.76.3:34681
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb | hidden | 127.31.76.1:35055
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:40977
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_0.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb | hidden | 127.31.76.3:34681
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:33665 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+---------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260812 09:41:22.714982 32048 log_verifier.cc:126] Checking tablet 1113e6c1419b4c58bfbd52945ef120f7
I20260812 09:41:22.834040 32048 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 1113e6c1419b4c58bfbd52945ef120f7
I20260812 09:41:22.834210 32048 log_verifier.cc:126] Checking tablet 1a8080e52199402ebd951934e7e04ce2
I20260812 09:41:22.968137 32048 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 1a8080e52199402ebd951934e7e04ce2
I20260812 09:41:22.968220 32048 log_verifier.cc:126] Checking tablet ce91946b5af8449ba2ec0160bf809d5a
I20260812 09:41:23.059406 32048 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ce91946b5af8449ba2ec0160bf809d5a
I20260812 09:41:23.060758 32048 test_util.cc:268] Using random seed: 1676551185
I20260812 09:41:23.073856 32048 all_types-itest.cc:600] Scanning tablet: [, int8 key=42)
I20260812 09:41:23.076285 32048 test_util.cc:268] Using random seed: 1676566714
I20260812 09:41:23.190644 32048 all_types-itest.cc:600] Scanning tablet: [int8 key=42, int8 key=84)
I20260812 09:41:23.193989 32048 test_util.cc:268] Using random seed: 1676684416
I20260812 09:41:23.199059 32048 all_types-itest.cc:600] Scanning tablet: [int8 key=84, )
I20260812 09:41:23.202564 32048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu with pid 32187
I20260812 09:41:23.218451 32048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu with pid 32376
I20260812 09:41:23.235180 32048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu with pid 32568
I20260812 09:41:23.251061 32048 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu with pid 32058
2026-08-12T09:41:23Z chronyd exiting
[ OK ] AllTypesItest/0.TestTimestampPadding (2347 ms)
[----------] 1 test from AllTypesItest/0 (2347 ms total)
[----------] 1 test from AllTypesItest/4, where TypeParam = kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> >
[ RUN ] AllTypesItest/4.TestTimestampPadding
I20260812 09:41:23.291424 32048 test_util.cc:268] Using random seed: 1676781849
2026-08-12T09:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T09:41:23Z Disabled control of system clock
I20260812 09:41:23.302436 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.76.62:44395
--webserver_interface=127.31.76.62
--webserver_port=0
--builtin_ntp_servers=127.31.76.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.76.62:44395 with env {}
W20260812 09:41:23.420753 417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:23.421031 417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:23.421066 417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:23.425189 417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 09:41:23.425254 417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:23.425272 417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 09:41:23.425289 417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 09:41:23.430164 417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.76.62:44395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.76.62:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.76.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:23.431386 417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:23.432942 417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:23.442194 426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.442198 422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.442194 423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:23.442443 417 server_base.cc:1034] running on GCE node
I20260812 09:41:23.443033 417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:23.444216 417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:23.445417 417 hybrid_clock.cc:648] HybridClock initialized: now 1786527683445395 us; error 35 us; skew 500 ppm
I20260812 09:41:23.447476 417 webserver.cc:460] Webserver started at http://127.31.76.62:40091/ using document root <none> and password file <none>
I20260812 09:41:23.448110 417 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:23.448176 417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:23.448432 417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:23.450491 417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/instance:
uuid: "48b548041a6b42c0a9d462dc50524b4c"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.451087 417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal/instance:
uuid: "48b548041a6b42c0a9d462dc50524b4c"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.454919 417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 09:41:23.457492 434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.458745 417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 09:41:23.458882 417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
uuid: "48b548041a6b42c0a9d462dc50524b4c"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.458990 417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:23.490262 417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:23.491093 417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:23.491338 417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:23.499284 417 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.62:44395
I20260812 09:41:23.499295 507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.62:44395 every 8 connection(s)
I20260812 09:41:23.500337 417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb
I20260812 09:41:23.503823 508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:23.509276 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 417
I20260812 09:41:23.509423 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/wal/instance
I20260812 09:41:23.509495 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: Bootstrap starting.
I20260812 09:41:23.512490 508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:23.513325 508 log.cc:824] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:23.515447 508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: No bootstrap required, opened a new log
I20260812 09:41:23.520251 508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } }
I20260812 09:41:23.520512 508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:23.520581 508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b548041a6b42c0a9d462dc50524b4c, State: Initialized, Role: FOLLOWER
I20260812 09:41:23.521066 508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } }
I20260812 09:41:23.521231 508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 09:41:23.521320 508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 09:41:23.521458 508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:23.522382 508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } }
I20260812 09:41:23.522689 508 leader_election.cc:302] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b548041a6b42c0a9d462dc50524b4c; no voters:
I20260812 09:41:23.522954 508 leader_election.cc:288] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 09:41:23.523097 515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:23.523391 515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [term 1 LEADER]: Becoming Leader. State: Replica: 48b548041a6b42c0a9d462dc50524b4c, State: Running, Role: LEADER
I20260812 09:41:23.523695 515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } }
I20260812 09:41:23.524255 508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 09:41:23.525501 517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b548041a6b42c0a9d462dc50524b4c. Latest consensus state: current_term: 1 leader_uuid: "48b548041a6b42c0a9d462dc50524b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } } }
I20260812 09:41:23.525647 517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [sys.catalog]: This master's current role is: LEADER
I20260812 09:41:23.525911 516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b548041a6b42c0a9d462dc50524b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b548041a6b42c0a9d462dc50524b4c" member_type: VOTER last_known_addr { host: "127.31.76.62" port: 44395 } } }
I20260812 09:41:23.526002 516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c [sys.catalog]: This master's current role is: LEADER
I20260812 09:41:23.526947 527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 09:41:23.529996 527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 09:41:23.535648 527 catalog_manager.cc:1347] Generated new cluster ID: 3ad4b41816114f4faed9d15df8f5405f
I20260812 09:41:23.535732 527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 09:41:23.546188 527 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 09:41:23.547567 527 catalog_manager.cc:1504] Loading token signing keys...
I20260812 09:41:23.554769 527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: Generated new TSK 0
I20260812 09:41:23.555651 527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 09:41:23.562175 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.1:0
--local_ip_for_outbound_sockets=127.31.76.1
--webserver_interface=127.31.76.1
--webserver_port=0
--tserver_master_addrs=127.31.76.62:44395
--builtin_ntp_servers=127.31.76.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
W20260812 09:41:23.683823 538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:23.684154 538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:23.684233 538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:23.689584 538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:23.689692 538 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:23.689731 538 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:23.689800 538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.1
I20260812 09:41:23.694864 538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.76.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:44395
--never_fsync=true
--heap_profile_path=/tmp/kudu.538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.1
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:23.696512 538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:23.698046 538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:23.708060 551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.708076 550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.708056 553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:23.708349 538 server_base.cc:1034] running on GCE node
I20260812 09:41:23.708853 538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:23.709620 538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:23.710816 538 hybrid_clock.cc:648] HybridClock initialized: now 1786527683710783 us; error 44 us; skew 500 ppm
I20260812 09:41:23.712704 538 webserver.cc:460] Webserver started at http://127.31.76.1:41105/ using document root <none> and password file <none>
I20260812 09:41:23.713326 538 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:23.713488 538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:23.713739 538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:23.715710 538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/instance:
uuid: "51bce1a9cf8949ae905ebcb8ddce105b"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.716306 538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal/instance:
uuid: "51bce1a9cf8949ae905ebcb8ddce105b"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.720299 538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 09:41:23.722978 559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.724260 538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 09:41:23.724411 538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
uuid: "51bce1a9cf8949ae905ebcb8ddce105b"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.724530 538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:23.739940 538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:23.740684 538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:23.740893 538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:23.741616 538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:23.742659 538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:23.742734 538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.742794 538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:23.742816 538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.752581 538 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.1:42099
I20260812 09:41:23.752621 726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.1:42099 every 8 connection(s)
I20260812 09:41:23.753607 538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb
I20260812 09:41:23.761014 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 538
I20260812 09:41:23.761157 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/wal/instance
I20260812 09:41:23.764063 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.2:0
--local_ip_for_outbound_sockets=127.31.76.2
--webserver_interface=127.31.76.2
--webserver_port=0
--tserver_master_addrs=127.31.76.62:44395
--builtin_ntp_servers=127.31.76.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 09:41:23.765385 727 heartbeater.cc:347] Connected to a master server at 127.31.76.62:44395
I20260812 09:41:23.765697 727 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:23.766400 727 heartbeater.cc:511] Master 127.31.76.62:44395 requested a full tablet report, sending...
I20260812 09:41:23.768014 455 ts_manager.cc:194] Registered new tserver with Master: 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099)
I20260812 09:41:23.769219 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.1:55759
W20260812 09:41:23.885144 730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:23.885484 730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:23.885571 730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:23.889719 730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:23.889828 730 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:23.889869 730 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:23.889938 730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.2
I20260812 09:41:23.894701 730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.76.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:44395
--never_fsync=true
--heap_profile_path=/tmp/kudu.730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.2
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:23.895872 730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:23.897403 730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:23.907868 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.907869 741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:23.907869 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:23.908356 730 server_base.cc:1034] running on GCE node
I20260812 09:41:23.908809 730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:23.909530 730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:23.910713 730 hybrid_clock.cc:648] HybridClock initialized: now 1786527683910692 us; error 32 us; skew 500 ppm
I20260812 09:41:23.912776 730 webserver.cc:460] Webserver started at http://127.31.76.2:35107/ using document root <none> and password file <none>
I20260812 09:41:23.913415 730 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:23.913525 730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:23.913739 730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:23.915706 730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/instance:
uuid: "08702823bf7945c699d002263093cd05"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.916275 730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal/instance:
uuid: "08702823bf7945c699d002263093cd05"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.920231 730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 09:41:23.922878 756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.924249 730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 09:41:23.924365 730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
uuid: "08702823bf7945c699d002263093cd05"
format_stamp: "Formatted at 2026-08-12 09:41:23 on dist-test-slave-bw88"
I20260812 09:41:23.924458 730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:23.949535 730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:23.950290 730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:23.950452 730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:23.951085 730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:23.952139 730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:23.952188 730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.952266 730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:23.952323 730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:23.961710 730 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.2:43897
I20260812 09:41:23.961812 920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.2:43897 every 8 connection(s)
I20260812 09:41:23.962813 730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb
I20260812 09:41:23.972865 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 730
I20260812 09:41:23.973006 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/wal/instance
I20260812 09:41:23.974097 923 heartbeater.cc:347] Connected to a master server at 127.31.76.62:44395
I20260812 09:41:23.974375 923 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:23.975021 923 heartbeater.cc:511] Master 127.31.76.62:44395 requested a full tablet report, sending...
I20260812 09:41:23.975478 32048 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
/tmp/dist-test-taskHr75Lv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.76.3:0
--local_ip_for_outbound_sockets=127.31.76.3
--webserver_interface=127.31.76.3
--webserver_port=0
--tserver_master_addrs=127.31.76.62:44395
--builtin_ntp_servers=127.31.76.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001
--consensus_max_batch_size_bytes=2097152 with env {}
I20260812 09:41:23.976119 455 ts_manager.cc:194] Registered new tserver with Master: 08702823bf7945c699d002263093cd05 (127.31.76.2:43897)
I20260812 09:41:23.976704 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.2:58367
W20260812 09:41:24.099483 928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 09:41:24.099753 928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 09:41:24.099794 928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 09:41:24.104048 928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 09:41:24.104116 928 flags.cc:425] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260812 09:41:24.104133 928 flags.cc:425] Enabled experimental flag: --flush_threshold_mb=1
W20260812 09:41:24.104169 928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.76.3
I20260812 09:41:24.108973 928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.76.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_max_batch_size_bytes=2097152
--fs_data_dirs=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.76.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.76.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--flush_threshold_mb=1
--tserver_master_addrs=127.31.76.62:44395
--never_fsync=true
--heap_profile_path=/tmp/kudu.928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.76.3
--log_dir=/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 08:09:26 UTC on ccycloud.quasar-kitgea.root.comops.site
build id 4582138
I20260812 09:41:24.110092 928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 09:41:24.111431 928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 09:41:24.121637 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:24.121627 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 09:41:24.121620 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 09:41:24.121804 928 server_base.cc:1034] running on GCE node
I20260812 09:41:24.122285 928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 09:41:24.122982 928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 09:41:24.124158 928 hybrid_clock.cc:648] HybridClock initialized: now 1786527684124141 us; error 26 us; skew 500 ppm
I20260812 09:41:24.126358 928 webserver.cc:460] Webserver started at http://127.31.76.3:41463/ using document root <none> and password file <none>
I20260812 09:41:24.126992 928 fs_manager.cc:359] Metadata directory not provided
I20260812 09:41:24.127058 928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 09:41:24.127331 928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 09:41:24.129634 928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/instance:
uuid: "fe00076da7784496bd55b00c1661bf13"
format_stamp: "Formatted at 2026-08-12 09:41:24 on dist-test-slave-bw88"
I20260812 09:41:24.130224 928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal/instance:
uuid: "fe00076da7784496bd55b00c1661bf13"
format_stamp: "Formatted at 2026-08-12 09:41:24 on dist-test-slave-bw88"
I20260812 09:41:24.134078 928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 09:41:24.136586 948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:24.138051 928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 09:41:24.138203 928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
uuid: "fe00076da7784496bd55b00c1661bf13"
format_stamp: "Formatted at 2026-08-12 09:41:24 on dist-test-slave-bw88"
I20260812 09:41:24.138326 928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 09:41:24.164006 928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 09:41:24.164778 928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 09:41:24.165056 928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 09:41:24.165841 928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 09:41:24.167097 928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 09:41:24.167182 928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:24.167294 928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 09:41:24.167330 928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 09:41:24.180114 928 rpc_server.cc:307] RPC server started. Bound to: 127.31.76.3:37683
I20260812 09:41:24.180132 1107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.76.3:37683 every 8 connection(s)
I20260812 09:41:24.181113 928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/info.pb
I20260812 09:41:24.184780 32048 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHr75Lv/build/debug/bin/kudu as pid 928
I20260812 09:41:24.184901 32048 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/wal/instance
I20260812 09:41:24.196985 1112 heartbeater.cc:347] Connected to a master server at 127.31.76.62:44395
I20260812 09:41:24.197232 1112 heartbeater.cc:464] Registering TS with master...
I20260812 09:41:24.197896 1112 heartbeater.cc:511] Master 127.31.76.62:44395 requested a full tablet report, sending...
I20260812 09:41:24.199199 455 ts_manager.cc:194] Registered new tserver with Master: fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
I20260812 09:41:24.199826 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.76.3:57605
I20260812 09:41:24.211290 32048 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 09:41:24.220355 455 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38892:
name: "all-types-table"
schema {
columns {
name: "key"
type: DECIMAL32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "int8_val"
type: INT8
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int16_val"
type: INT16
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int32_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int64_val"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "timestamp_val"
type: UNIXTIME_MICROS
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "date_val"
type: DATE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "varchar_val"
type: VARCHAR
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
length: 65535
}
immutable: false
}
columns {
name: "bool_val"
type: BOOL
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "float_val"
type: FLOAT
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "double_val"
type: DOUBLE
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "binary_val"
type: BINARY
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "decimal32_val"
type: DECIMAL32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 9
scale: 0
}
immutable: false
}
columns {
name: "decimal64_val"
type: DECIMAL64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 18
scale: 0
}
immutable: false
}
columns {
name: "decimal128_val"
type: DECIMAL128
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
type_attributes {
precision: 38
scale: 0
}
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\000\000\252\252\252*\004\001\000\000\000TUUU"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 09:41:24.224400 455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table all-types-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 09:41:24.245106 826 tablet_service.cc:1467] Processing CreateTablet for tablet 36288ce10b364d2dae07ac4e3706e40e (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260812 09:41:24.246909 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36288ce10b364d2dae07ac4e3706e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.246155 1009 tablet_service.cc:1467] Processing CreateTablet for tablet 36288ce10b364d2dae07ac4e3706e40e (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260812 09:41:24.246150 1007 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7e5067ea9743dd9942e27f4ea6c391 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260812 09:41:24.246327 1008 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29c709b2048d3a1eee19e376c8761 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260812 09:41:24.247783 1008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29c709b2048d3a1eee19e376c8761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.248147 1009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36288ce10b364d2dae07ac4e3706e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.247431 630 tablet_service.cc:1467] Processing CreateTablet for tablet 36288ce10b364d2dae07ac4e3706e40e (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION VALUES < 715827882_D32
I20260812 09:41:24.248850 629 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29c709b2048d3a1eee19e376c8761 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260812 09:41:24.249096 1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7e5067ea9743dd9942e27f4ea6c391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.249686 628 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7e5067ea9743dd9942e27f4ea6c391 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260812 09:41:24.250290 822 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7e5067ea9743dd9942e27f4ea6c391 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 1431655764_D32 <= VALUES
I20260812 09:41:24.250581 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7e5067ea9743dd9942e27f4ea6c391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.254765 629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29c709b2048d3a1eee19e376c8761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.255152 628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7e5067ea9743dd9942e27f4ea6c391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.245106 824 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd29c709b2048d3a1eee19e376c8761 (DEFAULT_TABLE table=all-types-table [id=08dff41a96564cde80deda7c44619ca2]), partition=RANGE (key) PARTITION 715827882_D32 <= VALUES < 1431655764_D32
I20260812 09:41:24.256219 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd29c709b2048d3a1eee19e376c8761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.256258 1139 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: Bootstrap starting.
I20260812 09:41:24.259291 1139 tablet_bootstrap.cc:654] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.260219 1139 log.cc:824] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:24.260739 630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36288ce10b364d2dae07ac4e3706e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 09:41:24.261250 1141 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Bootstrap starting.
I20260812 09:41:24.263689 1137 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: Bootstrap starting.
I20260812 09:41:24.264169 1141 tablet_bootstrap.cc:654] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.265053 1141 log.cc:824] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:24.266459 1137 tablet_bootstrap.cc:654] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.266980 1139 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: No bootstrap required, opened a new log
I20260812 09:41:24.267161 1141 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: No bootstrap required, opened a new log
I20260812 09:41:24.267238 1139 ts_tablet_manager.cc:1397] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260812 09:41:24.267361 1141 ts_tablet_manager.cc:1397] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 09:41:24.267328 1137 log.cc:824] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: Log is configured to *not* fsync() on all Append() calls
I20260812 09:41:24.269390 1137 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: No bootstrap required, opened a new log
I20260812 09:41:24.269575 1137 ts_tablet_manager.cc:1397] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 09:41:24.270681 1139 raft_consensus.cc:348] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.270948 1139 raft_consensus.cc:374] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.271015 1139 raft_consensus.cc:729] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08702823bf7945c699d002263093cd05, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.271241 1141 raft_consensus.cc:348] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.271498 1141 raft_consensus.cc:374] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.271565 1141 raft_consensus.cc:729] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bce1a9cf8949ae905ebcb8ddce105b, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.271521 1139 consensus_queue.cc:260] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.271991 1141 consensus_queue.cc:260] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.272241 923 heartbeater.cc:503] Master 127.31.76.62:44395 was elected leader, sending a full tablet report...
I20260812 09:41:24.272706 727 heartbeater.cc:503] Master 127.31.76.62:44395 was elected leader, sending a full tablet report...
I20260812 09:41:24.273002 1141 ts_tablet_manager.cc:1428] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 09:41:24.273219 1139 ts_tablet_manager.cc:1428] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 09:41:24.273523 1139 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05: Bootstrap starting.
I20260812 09:41:24.273828 1141 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: Bootstrap starting.
I20260812 09:41:24.274663 1139 tablet_bootstrap.cc:654] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.274801 1141 tablet_bootstrap.cc:654] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.276245 1137 raft_consensus.cc:348] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.276511 1137 raft_consensus.cc:374] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.276562 1137 raft_consensus.cc:729] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00076da7784496bd55b00c1661bf13, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.277009 1137 consensus_queue.cc:260] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.277740 1141 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: No bootstrap required, opened a new log
I20260812 09:41:24.277879 1141 ts_tablet_manager.cc:1397] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 09:41:24.278059 1137 ts_tablet_manager.cc:1428] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 09:41:24.278363 1112 heartbeater.cc:503] Master 127.31.76.62:44395 was elected leader, sending a full tablet report...
I20260812 09:41:24.278414 1137 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: Bootstrap starting.
I20260812 09:41:24.279026 1139 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05: No bootstrap required, opened a new log
I20260812 09:41:24.279105 1139 ts_tablet_manager.cc:1397] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 09:41:24.279428 1137 tablet_bootstrap.cc:654] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.279778 1139 raft_consensus.cc:348] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.279925 1139 raft_consensus.cc:374] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.279819 1141 raft_consensus.cc:348] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.279985 1139 raft_consensus.cc:729] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08702823bf7945c699d002263093cd05, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.280068 1141 raft_consensus.cc:374] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.280110 1139 consensus_queue.cc:260] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.280181 1141 raft_consensus.cc:729] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bce1a9cf8949ae905ebcb8ddce105b, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.280431 1139 ts_tablet_manager.cc:1428] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:41:24.280614 1139 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: Bootstrap starting.
I20260812 09:41:24.280573 1141 consensus_queue.cc:260] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.280750 1137 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: No bootstrap required, opened a new log
I20260812 09:41:24.280843 1137 ts_tablet_manager.cc:1397] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 09:41:24.281036 1141 ts_tablet_manager.cc:1428] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 09:41:24.281503 1137 raft_consensus.cc:348] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.281656 1137 raft_consensus.cc:374] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.281701 1137 raft_consensus.cc:729] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00076da7784496bd55b00c1661bf13, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.281869 1141 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b: Bootstrap starting.
I20260812 09:41:24.281889 1139 tablet_bootstrap.cc:654] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.281834 1137 consensus_queue.cc:260] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.282091 1137 ts_tablet_manager.cc:1428] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 09:41:24.282258 1137 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: Bootstrap starting.
I20260812 09:41:24.283033 1141 tablet_bootstrap.cc:654] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.283257 1137 tablet_bootstrap.cc:654] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: Neither blocks nor log segments found. Creating new log.
I20260812 09:41:24.284524 1141 tablet_bootstrap.cc:492] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b: No bootstrap required, opened a new log
I20260812 09:41:24.284610 1141 ts_tablet_manager.cc:1397] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:41:24.284920 1137 tablet_bootstrap.cc:492] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: No bootstrap required, opened a new log
I20260812 09:41:24.284989 1137 ts_tablet_manager.cc:1397] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 09:41:24.285223 1141 raft_consensus.cc:348] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.285364 1141 raft_consensus.cc:374] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.285403 1141 raft_consensus.cc:729] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bce1a9cf8949ae905ebcb8ddce105b, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.285472 1137 raft_consensus.cc:348] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.285544 1141 consensus_queue.cc:260] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.285591 1137 raft_consensus.cc:374] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.285640 1137 raft_consensus.cc:729] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00076da7784496bd55b00c1661bf13, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.285768 1137 consensus_queue.cc:260] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.285830 1141 ts_tablet_manager.cc:1428] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 09:41:24.286007 1137 ts_tablet_manager.cc:1428] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 09:41:24.286965 1139 tablet_bootstrap.cc:492] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: No bootstrap required, opened a new log
I20260812 09:41:24.287065 1139 ts_tablet_manager.cc:1397] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 09:41:24.287458 1139 raft_consensus.cc:348] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.287576 1139 raft_consensus.cc:374] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 09:41:24.287627 1139 raft_consensus.cc:729] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08702823bf7945c699d002263093cd05, State: Initialized, Role: FOLLOWER
I20260812 09:41:24.287810 1139 consensus_queue.cc:260] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.288079 1139 ts_tablet_manager.cc:1428] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 09:41:24.301409 1148 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:24.301666 1148 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.302918 1148 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08702823bf7945c699d002263093cd05 (127.31.76.2:43897), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
I20260812 09:41:24.307718 855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7e5067ea9743dd9942e27f4ea6c391" candidate_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08702823bf7945c699d002263093cd05" is_pre_election: true
I20260812 09:41:24.307734 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7e5067ea9743dd9942e27f4ea6c391" candidate_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13" is_pre_election: true
I20260812 09:41:24.308058 855 raft_consensus.cc:2454] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bce1a9cf8949ae905ebcb8ddce105b in term 0.
I20260812 09:41:24.308073 1039 raft_consensus.cc:2454] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bce1a9cf8949ae905ebcb8ddce105b in term 0.
I20260812 09:41:24.308621 561 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08702823bf7945c699d002263093cd05, 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.308908 1148 raft_consensus.cc:2781] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:24.309005 1148 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:24.309059 1148 raft_consensus.cc:3037] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.310179 1148 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.310549 1148 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [CANDIDATE]: Term 1 election: Requested vote from peers 08702823bf7945c699d002263093cd05 (127.31.76.2:43897), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
I20260812 09:41:24.310936 855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7e5067ea9743dd9942e27f4ea6c391" candidate_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08702823bf7945c699d002263093cd05"
I20260812 09:41:24.310940 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7e5067ea9743dd9942e27f4ea6c391" candidate_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13"
I20260812 09:41:24.311081 855 raft_consensus.cc:3037] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.311095 1039 raft_consensus.cc:3037] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.312234 1039 raft_consensus.cc:2454] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bce1a9cf8949ae905ebcb8ddce105b in term 1.
I20260812 09:41:24.312269 855 raft_consensus.cc:2454] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bce1a9cf8949ae905ebcb8ddce105b in term 1.
I20260812 09:41:24.312558 561 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.312760 1148 raft_consensus.cc:2781] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:24.313059 1148 raft_consensus.cc:686] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [term 1 LEADER]: Becoming Leader. State: Replica: 51bce1a9cf8949ae905ebcb8ddce105b, State: Running, Role: LEADER
I20260812 09:41:24.313521 1148 consensus_queue.cc:237] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.316920 459 catalog_manager.cc:5495] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1). New cstate: current_term: 1 leader_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } health_report { overall_health: UNKNOWN } } }
I20260812 09:41:24.396040 1145 raft_consensus.cc:482] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:24.396210 1145 raft_consensus.cc:504] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.397414 1145 leader_election.cc:288] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683), 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099)
I20260812 09:41:24.402050 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36288ce10b364d2dae07ac4e3706e40e" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13" is_pre_election: true
I20260812 09:41:24.402064 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36288ce10b364d2dae07ac4e3706e40e" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" is_pre_election: true
I20260812 09:41:24.402217 1039 raft_consensus.cc:2454] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 0.
I20260812 09:41:24.402374 659 raft_consensus.cc:2454] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 0.
I20260812 09:41:24.402640 760 leader_election.cc:302] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08702823bf7945c699d002263093cd05, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.402933 1145 raft_consensus.cc:2781] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:24.403012 1145 raft_consensus.cc:482] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:24.403059 1145 raft_consensus.cc:3037] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.404024 1145 raft_consensus.cc:504] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.404376 1145 leader_election.cc:288] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 election: Requested vote from peers fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683), 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099)
I20260812 09:41:24.404743 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36288ce10b364d2dae07ac4e3706e40e" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13"
I20260812 09:41:24.404795 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36288ce10b364d2dae07ac4e3706e40e" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bce1a9cf8949ae905ebcb8ddce105b"
I20260812 09:41:24.404855 1039 raft_consensus.cc:3037] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.404946 659 raft_consensus.cc:3037] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.405869 1039 raft_consensus.cc:2454] T 36288ce10b364d2dae07ac4e3706e40e P fe00076da7784496bd55b00c1661bf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 1.
I20260812 09:41:24.406010 659 raft_consensus.cc:2454] T 36288ce10b364d2dae07ac4e3706e40e P 51bce1a9cf8949ae905ebcb8ddce105b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 1.
I20260812 09:41:24.406275 760 leader_election.cc:302] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08702823bf7945c699d002263093cd05, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.406504 1145 raft_consensus.cc:2781] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:24.406761 1145 raft_consensus.cc:686] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [term 1 LEADER]: Becoming Leader. State: Replica: 08702823bf7945c699d002263093cd05, State: Running, Role: LEADER
I20260812 09:41:24.407131 1145 consensus_queue.cc:237] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } }
I20260812 09:41:24.410840 459 catalog_manager.cc:5495] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08702823bf7945c699d002263093cd05 (127.31.76.2). New cstate: current_term: 1 leader_uuid: "08702823bf7945c699d002263093cd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } health_report { overall_health: UNKNOWN } } }
W20260812 09:41:24.432438 1114 tablet.cc:2369] T 0e7e5067ea9743dd9942e27f4ea6c391 P fe00076da7784496bd55b00c1661bf13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:41:24.436529 1145 raft_consensus.cc:482] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 09:41:24.436698 1145 raft_consensus.cc:504] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.437529 1145 leader_election.cc:288] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
I20260812 09:41:24.438154 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29c709b2048d3a1eee19e376c8761" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13" is_pre_election: true
I20260812 09:41:24.438154 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29c709b2048d3a1eee19e376c8761" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" is_pre_election: true
I20260812 09:41:24.438333 659 raft_consensus.cc:2454] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 0.
I20260812 09:41:24.438333 1039 raft_consensus.cc:2454] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 0.
I20260812 09:41:24.438771 764 leader_election.cc:302] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08702823bf7945c699d002263093cd05, 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.439016 1145 raft_consensus.cc:2781] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 09:41:24.439095 1145 raft_consensus.cc:482] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 09:41:24.439127 1145 raft_consensus.cc:3037] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.440136 1145 raft_consensus.cc:504] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.440521 1145 leader_election.cc:288] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 election: Requested vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
I20260812 09:41:24.440927 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29c709b2048d3a1eee19e376c8761" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bce1a9cf8949ae905ebcb8ddce105b"
I20260812 09:41:24.441072 1039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd29c709b2048d3a1eee19e376c8761" candidate_uuid: "08702823bf7945c699d002263093cd05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe00076da7784496bd55b00c1661bf13"
I20260812 09:41:24.441115 659 raft_consensus.cc:3037] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.441250 1039 raft_consensus.cc:3037] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 09:41:24.442291 1039 raft_consensus.cc:2454] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 1.
I20260812 09:41:24.442314 659 raft_consensus.cc:2454] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08702823bf7945c699d002263093cd05 in term 1.
I20260812 09:41:24.442611 760 leader_election.cc:302] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08702823bf7945c699d002263093cd05, fe00076da7784496bd55b00c1661bf13; no voters:
I20260812 09:41:24.442858 1145 raft_consensus.cc:2781] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 09:41:24.442975 1145 raft_consensus.cc:686] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [term 1 LEADER]: Becoming Leader. State: Replica: 08702823bf7945c699d002263093cd05, State: Running, Role: LEADER
I20260812 09:41:24.443379 1145 consensus_queue.cc:237] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:24.445441 459 catalog_manager.cc:5495] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08702823bf7945c699d002263093cd05 (127.31.76.2). New cstate: current_term: 1 leader_uuid: "08702823bf7945c699d002263093cd05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } health_report { overall_health: UNKNOWN } } }
W20260812 09:41:24.464228 926 tablet.cc:2369] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 09:41:24.505023 728 tablet.cc:2369] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 09:41:24.692067 1148 consensus_queue.cc:1035] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:24.700260 1166 consensus_queue.cc:1035] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:24.839751 1145 consensus_queue.cc:1035] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:24.844317 1145 consensus_queue.cc:1035] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:24.936868 1145 consensus_queue.cc:1035] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:24.941082 1171 consensus_queue.cc:1035] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 09:41:26.186676 1114 maintenance_manager.cc:419] P fe00076da7784496bd55b00c1661bf13: Scheduling FlushMRSOp(1dd29c709b2048d3a1eee19e376c8761): perf score=1.000000
I20260812 09:41:26.218123 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling FlushMRSOp(1dd29c709b2048d3a1eee19e376c8761): perf score=1.937278
W20260812 09:41:26.404567 1116 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.31.76.2:43897, user_credentials={real_user=slave}} blocked reactor thread for 79643us
I20260812 09:41:26.509557 728 maintenance_manager.cc:419] P 51bce1a9cf8949ae905ebcb8ddce105b: Scheduling FlushMRSOp(1dd29c709b2048d3a1eee19e376c8761): perf score=1.937278
W20260812 09:41:26.621611 566 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/data (484864000 bytes available vs 518351011 bytes reserved) (error 28)
W20260812 09:41:26.630774 955 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-2/data/data (451301376 bytes available vs 518351011 bytes reserved) (error 28)
W20260812 09:41:26.641073 955 tablet_replica_mm_ops.cc:214] T 1dd29c709b2048d3a1eee19e376c8761 P fe00076da7784496bd55b00c1661bf13: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 1dd29c709b2048d3a1eee19e376c8761's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260812 09:41:26.645416 955 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 1dd29c709b2048d3a1eee19e376c8761's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f3cae3a29ad google::LogMessage::Fail() at ??:0
@ 0x7f3cae3a588a google::LogMessage::SendToLog() at ??:0
@ 0x7f3cae3a24de google::LogMessage::Flush() at ??:0
@ 0x7f3cae3a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3cb711b299 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f3cb0558674 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f3cb055573c _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f3cb055988c _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3cb05f43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3cb05f4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3cb05f63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3cb05e5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3cb1c296db start_thread at ??:0
@ 0x7f3cacea061f clone at ??:0
W20260812 09:41:27.160250 760 connection.cc:535] client connection to 127.31.76.3:37683 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:41:27.160519 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:41:27.161557 561 connection.cc:535] client connection to 127.31.76.3:37683 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:41:27.161767 561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:41:27.164063 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 09:41:27.164155 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 09:41:27.164739 561 consensus_peers.cc:487] T 0e7e5067ea9743dd9942e27f4ea6c391 P 51bce1a9cf8949ae905ebcb8ddce105b -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260812 09:41:27.270424 1121 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.31.76.1:42099, user_credentials={real_user=slave}} blocked reactor thread for 45987.2us
I20260812 09:41:27.327541 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 09:41:27.349471 630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260812 09:41:27.358825 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: FlushMRSOp(1dd29c709b2048d3a1eee19e376c8761) complete. Timing: real 1.140s user 0.645s sys 0.032s Metrics: {"bytes_written":472622,"cfile_init":1,"compiler_manager_pool.queue_time_us":6804,"compiler_manager_pool.run_cpu_time_us":301244,"compiler_manager_pool.run_wall_time_us":559834,"dirs.queue_time_us":349,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3871,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":392,"threads_started":3,"wal-append.queue_time_us":221}
I20260812 09:41:27.360641 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling UndoDeltaBlockGCOp(1dd29c709b2048d3a1eee19e376c8761): 62918 bytes on disk
I20260812 09:41:27.361567 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: UndoDeltaBlockGCOp(1dd29c709b2048d3a1eee19e376c8761) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20260812 09:41:27.362145 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling FlushMRSOp(0e7e5067ea9743dd9942e27f4ea6c391): perf score=1.937278
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | HEALTHY | <none> | 1 | 0
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897, 127.31.76.1:42099
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.1 | experimental | 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897, 127.31.76.1:42099
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897, 127.31.76.1:42099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_format | pb | hidden | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb | hidden | 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897, 127.31.76.1:42099
time_source | builtin | 127.31.76.2:43897, 127.31.76.1:42099
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897, tserver@127.31.76.1:42099
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:27.376793 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:27.591243 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 09:41:27.592044 630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | HEALTHY | <none> | 1 | 0
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897, 127.31.76.1:42099
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.1 | experimental | 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897, 127.31.76.1:42099
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897, 127.31.76.1:42099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_format | pb | hidden | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb | hidden | 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897, 127.31.76.1:42099
time_source | builtin | 127.31.76.2:43897, 127.31.76.1:42099
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897, tserver@127.31.76.1:42099
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:27.618793 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:27.783082 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: FlushMRSOp(0e7e5067ea9743dd9942e27f4ea6c391) complete. Timing: real 0.421s user 0.372s sys 0.008s Metrics: {"bytes_written":473407,"cfile_init":1,"dirs.queue_time_us":71,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2755,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000}
I20260812 09:41:27.783941 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling FlushMRSOp(36288ce10b364d2dae07ac4e3706e40e): perf score=1.937278
I20260812 09:41:27.915318 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 09:41:27.915660 630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | HEALTHY | <none> | 1 | 0
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897, 127.31.76.1:42099
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.1 | experimental | 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897, 127.31.76.1:42099
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897, 127.31.76.1:42099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_format | pb | hidden | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb | hidden | 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897, 127.31.76.1:42099
time_source | builtin | 127.31.76.2:43897, 127.31.76.1:42099
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897, tserver@127.31.76.1:42099
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:27.943769 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:28.196017 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: FlushMRSOp(36288ce10b364d2dae07ac4e3706e40e) complete. Timing: real 0.412s user 0.351s sys 0.004s Metrics: {"bytes_written":436486,"cfile_init":1,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":61094,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2575,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":10000,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":258}
I20260812 09:41:28.196794 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling UndoDeltaBlockGCOp(0e7e5067ea9743dd9942e27f4ea6c391): 63390 bytes on disk
I20260812 09:41:28.197302 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: UndoDeltaBlockGCOp(0e7e5067ea9743dd9942e27f4ea6c391) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20260812 09:41:28.197809 926 maintenance_manager.cc:419] P 08702823bf7945c699d002263093cd05: Scheduling UndoDeltaBlockGCOp(36288ce10b364d2dae07ac4e3706e40e): 62918 bytes on disk
I20260812 09:41:28.198174 766 maintenance_manager.cc:643] P 08702823bf7945c699d002263093cd05: UndoDeltaBlockGCOp(36288ce10b364d2dae07ac4e3706e40e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20260812 09:41:28.351121 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20260812 09:41:28.351933 630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | HEALTHY | <none> | 1 | 0
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+--------------------------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897, 127.31.76.1:42099
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.1 | experimental | 127.31.76.1:42099
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897, 127.31.76.1:42099
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897, 127.31.76.1:42099
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_format | pb | hidden | 127.31.76.2:43897, 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-0/data/info.pb | hidden | 127.31.76.1:42099
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897, 127.31.76.1:42099
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+--------------------------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897, 127.31.76.1:42099
time_source | builtin | 127.31.76.2:43897, 127.31.76.1:42099
Version Summary
Version | Servers
-----------------+---------------------------------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897, tserver@127.31.76.1:42099
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is under-replicated: 1 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): RUNNING [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+------------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNDER_REPLICATED | 3 | 0 | 0 | 3 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 1 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 1 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 1 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:28.379639 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:28.482324 566 tablet_replica_mm_ops.cc:214] T 1dd29c709b2048d3a1eee19e376c8761 P 51bce1a9cf8949ae905ebcb8ddce105b: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 1dd29c709b2048d3a1eee19e376c8761: unable to open output file for column decimal32_val DECIMAL(9, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 1dd29c709b2048d3a1eee19e376c8761's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260812 09:41:28.482483 566 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 1dd29c709b2048d3a1eee19e376c8761: unable to open output file for column decimal32_val DECIMAL(9, 0) NULLABLE: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 1dd29c709b2048d3a1eee19e376c8761's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f5f75f699ad google::LogMessage::Fail() at ??:0
@ 0x7f5f75f6c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5f75f694de google::LogMessage::Flush() at ??:0
@ 0x7f5f75f6aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f7ece2299 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f5f7811f674 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f5f7811c73c _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f5f7812088c _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5f781bb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5f781bbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5f781bd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5f781ac512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f797f06db start_thread at ??:0
@ 0x7f5f74a6761f clone at ??:0
W20260812 09:41:28.700677 764 connection.cc:535] client connection to 127.31.76.1:42099 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 09:41:28.701736 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 09:41:28.958356 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 09:41:28.965796 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:28.979435 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:29.052173 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260812 09:41:29.262001 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 09:41:29.516465 1483 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:29.516618 1483 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:29.517078 1483 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:29.517971 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:29.518296 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:29.518400 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:29.518543 1483 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:29.794773 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:29.814800 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:30.870432 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:30.891119 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:30.933092 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260812 09:41:31.059325 1483 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:31.059496 1483 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:31.060081 1483 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:31.061507 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:31.061722 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:31.061815 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:31.061981 1483 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:41:31.303117 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260812 09:41:31.462284 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260812 09:41:31.947300 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:31.967481 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:31.971979 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260812 09:41:32.369777 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20260812 09:41:32.750967 1483 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:32.751196 1483 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:32.751683 1483 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:32.753024 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:32.753122 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:32.753188 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:32.753387 1483 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:33.022866 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:33.043243 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:33.541932 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260812 09:41:33.640643 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260812 09:41:33.952373 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260812 09:41:34.096674 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:34.117035 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:34.244156 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260812 09:41:35.176268 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:35.195479 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:36.011121 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260812 09:41:36.224053 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260812 09:41:36.253885 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:36.274411 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:36.373993 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260812 09:41:36.455473 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260812 09:41:37.246760 1483 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:37.246941 1483 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:37.247462 1483 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:37.248696 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:37.248916 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:37.248972 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:37.249135 1483 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:37.329854 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:37.350381 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:37.385104 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 45 similar messages]
I20260812 09:41:38.406355 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:38.426287 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:38.596798 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260812 09:41:38.728672 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260812 09:41:38.791489 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260812 09:41:38.947947 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260812 09:41:39.501497 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:39.522015 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:40.071314 1483 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:40.071483 1483 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:40.071919 1483 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:40.073304 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:41:40.073658 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:41:40.073752 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:40.073908 1483 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:40.587011 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:40.607623 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:41.015863 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260812 09:41:41.352759 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260812 09:41:41.424443 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260812 09:41:41.521628 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260812 09:41:41.664530 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:41.686394 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:42.404613 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260812 09:41:42.742837 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:42.764254 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:43.460217 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260812 09:41:43.822021 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:43.841372 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:43.937366 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260812 09:41:44.002825 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260812 09:41:44.038441 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260812 09:41:44.902227 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:44.924439 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:45.869072 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260812 09:41:45.988754 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:46.008054 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:46.376497 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260812 09:41:46.415761 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260812 09:41:46.516075 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260812 09:41:46.731348 2062 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:46.731525 2062 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:46.732156 2062 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:46.733441 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:46.733650 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:46.733736 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:46.733943 2062 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:47.065975 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:47.086510 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:47.475898 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:41:48.144088 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:48.164045 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:48.337271 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260812 09:41:48.799868 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260812 09:41:48.910179 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260812 09:41:48.980469 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260812 09:41:49.221194 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:49.240424 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:50.302667 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:50.324119 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:51.006361 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260812 09:41:51.084911 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260812 09:41:51.103431 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260812 09:41:51.390309 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:51.410292 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:51.476454 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260812 09:41:52.468870 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 09:41:52.477214 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:52.490538 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:53.432029 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260812 09:41:53.556346 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:53.579257 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:53.598191 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260812 09:41:53.721720 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260812 09:41:53.937000 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260812 09:41:54.644701 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:54.665868 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:41:55.444972 2062 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:41:55.445173 2062 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:41:55.445911 2062 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:41:55.447245 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:41:55.447445 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:41:55.447551 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:41:55.447683 2062 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:41:55.727368 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:55.748833 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:55.854681 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260812 09:41:56.030944 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260812 09:41:56.182344 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260812 09:41:56.386600 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260812 09:41:56.835067 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:56.856952 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:57.571965 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:41:57.916164 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:57.936502 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:58.386435 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260812 09:41:58.749261 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260812 09:41:58.953600 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260812 09:41:59.001784 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:41:59.021880 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:41:59.160403 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260812 09:42:00.078462 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:00.098359 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:00.730350 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260812 09:42:01.104358 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260812 09:42:01.158989 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
W20260812 09:42:01.179287 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:01.181102 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:01.766144 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260812 09:42:02.242942 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:02.262563 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:02.727918 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:42:03.152931 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260812 09:42:03.322302 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:03.342496 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:03.597916 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260812 09:42:03.990744 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260812 09:42:04.343271 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260812 09:42:04.396402 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:04.416867 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:05.469625 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:05.487975 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:05.606932 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260812 09:42:05.900123 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260812 09:42:06.529709 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260812 09:42:06.543092 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:06.561870 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:06.835005 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260812 09:42:07.620920 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:07.641263 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:07.767350 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:42:07.767553 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:42:07.768105 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:42:07.768960 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:42:07.769718 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:42:07.769834 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:42:07.769928 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:42:07.770104 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:42:08.270569 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260812 09:42:08.418468 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260812 09:42:08.697960 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:08.718634 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:08.869445 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260812 09:42:09.206497 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20260812 09:42:09.790130 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:09.816321 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:10.872510 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:10.895013 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:10.998265 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260812 09:42:11.042001 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260812 09:42:11.100904 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260812 09:42:11.676728 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20260812 09:42:11.950814 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:11.973999 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:12.789937 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:42:13.035749 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:13.057731 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:13.318984 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20260812 09:42:13.580437 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260812 09:42:13.698441 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260812 09:42:13.705263 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:42:13.705446 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:42:13.705928 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:42:13.707000 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:42:13.707166 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:42:13.707230 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:42:13.707418 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:42:14.113587 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:14.132272 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:14.224834 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20260812 09:42:15.187521 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:15.207718 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:15.773124 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260812 09:42:16.163784 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20260812 09:42:16.199021 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260812 09:42:16.271134 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:16.292356 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:16.894934 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:42:16.895131 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:42:16.895680 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:42:16.897126 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:42:16.897271 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:42:16.897384 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:42:16.897584 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:42:17.112540 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260812 09:42:17.349100 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:17.369199 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:17.853752 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20260812 09:42:18.426690 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 09:42:18.431578 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:18.446784 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:18.875190 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20260812 09:42:18.958895 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260812 09:42:19.505263 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:19.525066 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:19.920102 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20260812 09:42:20.582911 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:20.603075 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:20.838670 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260812 09:42:21.275919 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20260812 09:42:21.525637 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260812 09:42:21.660887 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:21.682025 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:22.365404 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20260812 09:42:22.744197 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:22.764561 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:22.941028 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:42:23.276669 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260812 09:42:23.799367 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260812 09:42:23.807708 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260812 09:42:23.827057 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:23.847960 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:24.811188 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20260812 09:42:24.908080 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:24.929158 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:25.768857 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260812 09:42:25.986698 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:26.007179 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:26.147881 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20260812 09:42:26.351358 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20260812 09:42:27.064670 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:27.084493 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:27.284565 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20260812 09:42:28.029229 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20260812 09:42:28.145249 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:28.165162 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:28.370250 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260812 09:42:28.523178 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260812 09:42:28.569044 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260812 09:42:28.921579 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:42:28.921765 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:42:28.922403 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:42:28.923635 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:42:28.923765 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:42:28.923851 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:42:28.924050 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:42:29.228405 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:29.248709 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:29.879070 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20260812 09:42:30.306668 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:30.327076 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:30.989262 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260812 09:42:31.073426 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20260812 09:42:31.160398 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260812 09:42:31.386925 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:31.407850 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:32.428172 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20260812 09:42:32.466747 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:32.486016 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:33.427734 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:42:33.428618 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260812 09:42:33.444082 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20260812 09:42:33.469635 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260812 09:42:33.541793 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:33.562384 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:34.628756 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:34.648470 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:35.020753 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20260812 09:42:35.708302 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:35.728116 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:35.807873 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260812 09:42:35.918000 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20260812 09:42:36.075971 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260812 09:42:36.788017 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:36.808840 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:37.448580 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20260812 09:42:37.865764 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:37.887506 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:38.264983 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 09:42:38.295914 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 09:42:38.555757 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:42:38.556674 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20260812 09:42:38.947515 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:38.968050 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:40.032177 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:40.052340 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:40.192543 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20260812 09:42:40.644344 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260812 09:42:40.688023 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260812 09:42:41.032407 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20260812 09:42:41.114569 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:41.134886 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:42.195081 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:42.216120 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:42.773759 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20260812 09:42:43.150872 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260812 09:42:43.195807 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260812 09:42:43.202888 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20260812 09:42:43.278270 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:43.299261 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:43.608232 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260812 09:42:44.362504 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:44.384797 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:45.450858 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:45.471045 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:45.543661 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260812 09:42:45.604192 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260812 09:42:45.638176 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20260812 09:42:45.774868 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20260812 09:42:46.497468 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:42:46.497650 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:42:46.498131 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:42:46.499287 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:42:46.499388 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:42:46.499504 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:42:46.499682 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 09:42:46.529976 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:46.549675 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:47.611406 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:47.631915 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:47.919546 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260812 09:42:48.011199 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260812 09:42:48.049502 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260812 09:42:48.167484 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20260812 09:42:48.613639 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20260812 09:42:48.692109 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:48.711459 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:49.775628 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:49.795352 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:50.303249 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20260812 09:42:50.570899 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20260812 09:42:50.605315 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260812 09:42:50.627019 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260812 09:42:50.859789 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:50.880604 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:51.937208 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:51.956748 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:52.741590 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20260812 09:42:53.017073 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:53.040624 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:53.141535 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260812 09:42:53.192801 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260812 09:42:53.200811 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20260812 09:42:53.624778 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20260812 09:42:54.097626 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:54.120102 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:55.178915 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:55.200798 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:55.611869 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260812 09:42:55.641466 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260812 09:42:55.642199 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20260812 09:42:55.668771 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20260812 09:42:56.265861 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:56.287125 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:42:57.346660 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:57.369560 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:58.070457 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260812 09:42:58.115753 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260812 09:42:58.177215 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20260812 09:42:58.431332 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:58.452246 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:42:58.540205 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260812 09:42:58.748684 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260812 09:42:59.517532 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:42:59.537407 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:00.542691 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260812 09:43:00.636582 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 09:43:00.647751 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:00.670100 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:00.746677 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20260812 09:43:00.808768 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20260812 09:43:01.733270 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:01.754199 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:02.811264 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:02.831573 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:02.980290 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20260812 09:43:02.996888 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260812 09:43:03.064906 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260812 09:43:03.304301 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260812 09:43:03.850237 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260812 09:43:03.890096 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:03.912511 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:04.970249 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:04.990655 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:05.471727 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:43:05.471905 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:43:05.472471 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:43:05.473871 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:43:05.473987 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:43:05.474072 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:43:05.474239 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:43:05.544953 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20260812 09:43:05.583575 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260812 09:43:05.673440 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20260812 09:43:05.800495 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260812 09:43:06.052028 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:06.073019 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:07.138731 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:07.158989 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:08.046686 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260812 09:43:08.104152 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20260812 09:43:08.170853 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20260812 09:43:08.215757 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:08.236200 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:08.346880 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260812 09:43:08.856004 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:43:09.291182 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:09.310549 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:10.366710 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:10.386042 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:10.428391 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20260812 09:43:10.497030 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260812 09:43:10.679517 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260812 09:43:10.691756 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20260812 09:43:11.443957 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:11.464886 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:12.527714 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:12.547616 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:13.040951 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260812 09:43:13.041872 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20260812 09:43:13.096881 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260812 09:43:13.410564 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20260812 09:43:13.606698 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:13.626807 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:13.995553 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260812 09:43:14.687656 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:14.708089 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:15.470409 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260812 09:43:15.572146 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20260812 09:43:15.766682 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20260812 09:43:15.776384 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:15.787943 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:15.971158 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20260812 09:43:16.845194 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:16.865836 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:17.932461 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:17.952569 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:18.031744 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260812 09:43:18.034873 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260812 09:43:18.253998 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20260812 09:43:18.270155 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20260812 09:43:19.010886 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:19.030876 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:19.153697 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:43:20.087739 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:20.107461 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:20.374030 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260812 09:43:20.579931 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260812 09:43:20.885564 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20260812 09:43:20.999692 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260812 09:43:21.162758 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:21.184597 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260812 09:43:22.247129 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:22.267278 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:22.753021 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 09:43:23.191319 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20260812 09:43:23.337848 822 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
48b548041a6b42c0a9d462dc50524b4c | 127.31.76.62:44395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.31.76.20:39083 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+-------------+----------+----------------+-----------------
08702823bf7945c699d002263093cd05 | 127.31.76.2:43897 | HEALTHY | <none> | 2 | 0
51bce1a9cf8949ae905ebcb8ddce105b | 127.31.76.1:42099 | UNAVAILABLE | <none> | n/a | n/a
fe00076da7784496bd55b00c1661bf13 | 127.31.76.3:37683 | UNAVAILABLE | <none> | n/a | n/a
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Error from 127.31.76.1:42099: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) (UNAVAILABLE)
Error from 127.31.76.3:37683: Network error: could not get status from server: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) (UNAVAILABLE)
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------
flush_threshold_mb | 1 | runtime,experimental | 127.31.76.2:43897
ipki_server_key_size | 768 | experimental | 127.31.76.2:43897
local_ip_for_outbound_sockets | 127.31.76.2 | experimental | 127.31.76.2:43897
never_fsync | true | unsafe,advanced | 127.31.76.2:43897
openssl_security_level_override | 0 | unsafe,hidden | 127.31.76.2:43897
rpc_server_allow_ephemeral_ports | true | unsafe | 127.31.76.2:43897
server_dump_info_format | pb | hidden | 127.31.76.2:43897
server_dump_info_path | /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/ts-1/data/info.pb | hidden | 127.31.76.2:43897
tablet_delta_store_major_compact_min_ratio | 0.001 | experimental | 127.31.76.2:43897
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------
builtin_ntp_servers | 127.31.76.20:39083 | 127.31.76.2:43897
time_source | builtin | 127.31.76.2:43897
Version Summary
Version | Servers
-----------------+------------------------------------------------------
1.18.0-SNAPSHOT | master@127.31.76.62:44395, tserver@127.31.76.2:43897
Tablet Summary
Tablet 36288ce10b364d2dae07ac4e3706e40e of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
All reported replicas are:
A = 08702823bf7945c699d002263093cd05
B = fe00076da7784496bd55b00c1661bf13
C = 51bce1a9cf8949ae905ebcb8ddce105b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | [config not available] | | |
Tablet 1dd29c709b2048d3a1eee19e376c8761 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING [LEADER]
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | [config not available] | | |
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
Tablet 0e7e5067ea9743dd9942e27f4ea6c391 of table 'all-types-table' is unavailable: 2 replica(s) not RUNNING
51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): TS unavailable [LEADER]
08702823bf7945c699d002263093cd05 (127.31.76.2:43897): RUNNING
fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): TS unavailable
All reported replicas are:
A = 51bce1a9cf8949ae905ebcb8ddce105b
B = 08702823bf7945c699d002263093cd05
C = fe00076da7784496bd55b00c1661bf13
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------------+----+-------------+---------------+---------+------------+------------------+-------------
all-types-table | 3 | UNAVAILABLE | 3 | 0 | 0 | 0 | 3
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 3
Replicas | 9
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
tserver unusual flags check error: 2 of 3 tservers were not available to retrieve unusual flags
tserver diverged flags check error: 2 of 3 tservers were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from tablet servers: failed to gather info for all tablet servers: 2 of 3 had errors
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260812 09:43:23.358772 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260812 09:43:23.389798 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20260812 09:43:23.526518 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20260812 09:43:23.692303 444 maintenance_manager.cc:419] P 48b548041a6b42c0a9d462dc50524b4c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033380
W20260812 09:43:23.695288 443 dir_manager.cc:174] Insufficient disk space under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/data: will retry after 10 seconds: IO error: Insufficient disk space to allocate 0 bytes under path /tmp/dist-test-taskHr75Lv/test-tmp/all_types-itest.1.AllTypesItest_4.TestTimestampPadding.1786527680882199-32048-0/minicluster-data/master-0/data/data (448225280 bytes available vs 518351011 bytes reserved) (error 28)
W20260812 09:43:23.695820 443 tablet_replica_mm_ops.cc:214] T 00000000000000000000000000000000 P 48b548041a6b42c0a9d462dc50524b4c: failed to flush MRS: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
F20260812 09:43:23.695901 443 tablet_replica_mm_ops.cc:215] Check failed: tablet->HasBeenStopped() Unrecoverable flush failure caused by error: IO error: Failed to open DiskRowSet for flush: tablet 00000000000000000000000000000000: unable to open output file for column entry_type INT8 NOT NULL: No directories could be added (1 dirs total, 0 failed, 1 full): IO error: No directories available in 00000000000000000000000000000000's directory group (1 dirs total, 0 failed, 1 full) (error 28) (error 19)
*** Check failure stack trace: ***
@ 0x7f011cb0c9ad google::LogMessage::Fail() at ??:0
@ 0x7f011cb0f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f011cb0c4de google::LogMessage::Flush() at ??:0
@ 0x7f011cb0da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0125885299 kudu::tablet::FlushMRSOp::Perform() at ??:0
@ 0x7f011ecc2674 kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7f011ecbf73c _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7f011ecc388c _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f011ed5e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f011ed5ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f011ed603f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f011ed4f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f01203936db start_thread at ??:0
@ 0x7f011b60a61f clone at ??:0
W20260812 09:43:24.154675 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260812 09:43:24.516268 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:43:24.516487 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:43:24.517094 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:43:24.518500 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:43:24.518646 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:43:24.518749 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:43:24.518914 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:43:24.697055 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:43:25.093998 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 09:43:25.711429 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 09:43:25.983515 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20260812 09:43:26.232679 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 09:43:27.589578 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 09:43:28.213020 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 09:43:28.459066 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20260812 09:43:28.879379 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 09:43:29.162570 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:43:30.169067 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 09:43:30.955027 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20260812 09:43:30.966497 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 09:43:31.506060 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 09:43:32.761754 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 09:43:33.356691 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20260812 09:43:33.502959 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 09:43:34.175990 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:43:34.176802 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 09:43:35.183261 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 09:43:35.914237 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20260812 09:43:36.382766 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 09:43:36.811226 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 09:43:37.702464 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 09:43:38.529413 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20260812 09:43:38.958968 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 09:43:39.190716 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:43:39.191484 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 09:43:40.046375 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 09:43:41.102496 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20260812 09:43:41.307139 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 09:43:41.863608 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 09:43:42.761411 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20260812 09:43:43.100749 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:43:43.100927 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:43:43.101562 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:43:43.102766 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:43:43.102902 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:43:43.102991 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:43:43.103163 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:43:43.782716 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20260812 09:43:43.858851 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 09:43:44.249130 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:43:44.384217 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 09:43:45.389700 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 09:43:46.324491 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20260812 09:43:46.395180 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 09:43:46.824779 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 09:43:47.565222 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 09:43:48.675386 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20260812 09:43:49.069777 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20260812 09:43:49.478358 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:43:49.493552 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 09:43:50.248409 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 09:43:51.016098 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20260812 09:43:51.065146 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:43:51.065397 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:43:51.065958 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:43:51.067305 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:43:51.067410 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:43:51.067529 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:43:51.067744 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:43:51.817201 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 09:43:52.154141 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 09:43:52.855022 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 09:43:53.513489 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20260812 09:43:54.210103 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 09:43:54.485885 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:43:54.900682 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 09:43:55.349813 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 09:43:56.022789 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20260812 09:43:56.743211 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 09:43:57.284813 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 09:43:58.025050 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 09:43:58.520789 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20260812 09:43:59.559638 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260812 09:43:59.560460 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 09:44:00.024987 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 09:44:00.606549 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 09:44:01.028609 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20260812 09:44:01.901537 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20260812 09:44:02.197899 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 09:44:03.096999 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 09:44:03.760483 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20260812 09:44:04.159906 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:44:04.160048 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:44:04.160526 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:44:04.161546 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:44:04.161773 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:44:04.161859 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:44:04.161996 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:44:04.575102 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:44:04.576073 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 09:44:04.714672 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 09:44:05.623353 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 09:44:05.934382 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20260812 09:44:07.058223 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 09:44:07.217315 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 09:44:08.074103 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 09:44:08.649101 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20260812 09:44:09.479257 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20260812 09:44:09.492786 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:44:09.492944 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:44:09.493498 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:44:09.494779 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:44:09.494894 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:44:09.495008 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:44:09.495193 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:44:09.602174 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:44:09.602988 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 09:44:10.618202 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 09:44:11.207211 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20260812 09:44:11.823485 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 09:44:11.839226 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 09:44:13.194298 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 09:44:13.757948 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20260812 09:44:13.965952 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 09:44:14.063910 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 09:44:14.626128 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:44:15.609179 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 09:44:16.155149 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 09:44:16.326104 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20260812 09:44:16.778218 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 09:44:17.939105 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 09:44:18.440150 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 09:44:18.693702 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20260812 09:44:19.173552 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 09:44:19.752256 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:44:20.578373 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 09:44:20.787204 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 09:44:21.206064 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20260812 09:44:21.769837 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 09:44:22.972229 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 09:44:23.233127 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20260812 09:44:23.711412 2593 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:44:23.711675 2593 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:44:23.712167 2593 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:44:23.713400 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:44:23.713546 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:44:23.713621 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:44:23.713804 2593 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:44:23.782845 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20260812 09:44:24.424738 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:44:24.521274 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:44:24.764655 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:44:24.771605 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (60 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:44:24.978181 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 09:44:25.809830 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 09:44:26.307158 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20260812 09:44:26.961773 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 09:44:27.516090 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 09:44:28.341379 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 09:44:29.016064 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20260812 09:44:29.316757 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 09:44:29.778491 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:44:29.896291 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 09:44:30.519982 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 09:44:31.334151 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20260812 09:44:31.784561 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 09:44:32.268389 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 09:44:32.920666 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 09:44:33.605017 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20260812 09:44:34.008656 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 09:44:34.567400 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 09:44:34.961695 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:44:35.591008 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 09:44:36.046072 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20260812 09:44:36.725006 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 09:44:36.936172 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20260812 09:44:38.041076 5463 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:44:38.041325 5463 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:44:38.041923 5463 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:44:38.043198 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:44:38.043316 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:44:38.043423 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:44:38.043635 5463 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:44:38.407903 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 09:44:38.468477 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20260812 09:44:39.414276 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 09:44:39.433437 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 09:44:39.968847 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:44:40.696444 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 09:44:40.902773 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20260812 09:44:42.100780 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 09:44:42.240835 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 09:44:43.306289 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20260812 09:44:43.479156 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 09:44:44.594247 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 09:44:44.624531 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20260812 09:44:45.024224 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260812 09:44:45.788563 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20260812 09:44:45.887880 5463 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:44:45.888068 5463 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:44:45.888736 5463 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:44:45.890228 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:44:45.890349 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:44:45.890450 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:44:45.890646 5463 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:44:46.083623 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 09:44:46.879079 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 09:44:47.124393 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 09:44:48.288177 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20260812 09:44:48.664634 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 09:44:49.214371 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 09:44:49.564080 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 09:44:50.065479 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:44:50.904443 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20260812 09:44:51.121199 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 09:44:51.656301 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 09:44:52.054479 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 09:44:53.420359 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20260812 09:44:53.500001 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 09:44:54.090590 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 09:44:54.802174 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 09:44:55.181262 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:44:55.953907 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20260812 09:44:56.149199 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 09:44:56.587188 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 09:44:57.018003 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 09:44:58.440263 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20260812 09:44:58.658528 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 09:44:59.045512 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 09:44:59.469425 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 09:45:00.251287 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:45:00.851387 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20260812 09:45:01.074097 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 09:45:01.738270 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 09:45:02.064180 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20260812 09:45:02.519083 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:02.519313 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:02.519953 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:02.521229 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:45:02.521401 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:45:02.521574 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:02.521772 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:03.398312 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 09:45:03.503950 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20260812 09:45:04.292286 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 09:45:04.758116 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 09:45:05.618587 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:45:05.619372 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20260812 09:45:05.638867 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 09:45:06.478426 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 09:45:07.309041 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 09:45:08.061111 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20260812 09:45:08.129007 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 09:45:09.199800 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 09:45:09.656857 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 09:45:10.367724 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20260812 09:45:10.658105 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:45:10.809501 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 09:45:11.663769 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 09:45:12.167732 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 09:45:12.809592 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20260812 09:45:13.089197 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 09:45:14.310374 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 09:45:14.659878 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 09:45:15.377009 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20260812 09:45:15.464465 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20260812 09:45:15.516345 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:15.516572 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:15.517154 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:15.518599 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:45:15.518754 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:45:15.518857 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:15.519080 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:15.819031 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260812 09:45:16.596503 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 09:45:17.245393 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 09:45:17.690013 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20260812 09:45:17.795405 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 09:45:18.914616 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 09:45:19.571000 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 09:45:20.140348 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20260812 09:45:20.170950 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 09:45:21.064424 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:45:21.517730 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 09:45:22.057081 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 09:45:22.673656 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20260812 09:45:22.813668 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 09:45:23.926548 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 09:45:24.349124 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 09:45:24.844244 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (120 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:45:25.194001 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20260812 09:45:25.209038 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:45:25.719890 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:45:26.236358 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:45:26.286789 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 09:45:26.955612 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 09:45:27.484102 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20260812 09:45:27.705006 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:27.705190 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:27.705744 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:27.707106 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:45:27.707255 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:45:27.707340 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:27.707513 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:27.822412 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20260812 09:45:28.762962 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 09:45:29.567230 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 09:45:30.205454 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 09:45:30.229787 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20260812 09:45:31.074100 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 09:45:31.255174 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:45:32.057922 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 09:45:32.473121 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20260812 09:45:32.888490 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 09:45:33.613018 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 09:45:34.592724 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 09:45:34.900913 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20260812 09:45:35.457676 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 09:45:36.193495 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 09:45:36.620033 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:45:37.168335 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20260812 09:45:37.250558 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20260812 09:45:37.481897 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:37.482080 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:37.482527 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:37.484078 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:45:37.484187 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:45:37.484252 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:37.484412 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:37.921229 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 09:45:38.687374 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 09:45:39.726305 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 09:45:39.776870 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20260812 09:45:40.721541 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 09:45:41.126777 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 09:45:41.678967 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:45:42.325246 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20260812 09:45:42.371505 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 09:45:43.108982 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 09:45:43.345759 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 09:45:44.639269 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20260812 09:45:44.807788 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 09:45:45.466696 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 09:45:45.589383 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 09:45:46.888114 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:45:46.888877 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20260812 09:45:47.001720 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 09:45:48.124676 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 09:45:48.144615 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 09:45:49.254454 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 09:45:49.322166 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20260812 09:45:50.542766 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 09:45:50.573508 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 09:45:51.622994 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20260812 09:45:51.637281 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 09:45:52.039626 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:45:53.173622 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 09:45:53.176889 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 09:45:53.777117 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20260812 09:45:54.448076 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20260812 09:45:55.420832 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:55.421043 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:55.421684 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:55.423014 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:45:55.423142 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:45:55.423255 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:55.423447 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:55.661989 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 09:45:55.698890 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260812 09:45:56.284524 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20260812 09:45:57.037993 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 09:45:57.202196 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:45:58.046541 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 09:45:58.249599 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20260812 09:45:58.342871 5468 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:45:58.343112 5468 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:45:58.343750 5468 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:45:58.345048 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:45:58.345263 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:45:58.345319 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:45:58.345520 5468 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:45:58.635388 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20260812 09:45:59.156497 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 09:46:00.551015 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 09:46:00.556303 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 09:46:01.013386 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20260812 09:46:01.477011 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 09:46:02.211356 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:46:03.171228 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 09:46:03.340859 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 09:46:03.422464 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20260812 09:46:03.659909 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 09:46:05.542186 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 09:46:05.697641 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 09:46:05.816358 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20260812 09:46:06.566352 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 09:46:07.301326 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:46:07.971411 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 09:46:08.339169 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 09:46:08.586630 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20260812 09:46:08.917179 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 09:46:10.217680 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 09:46:10.583153 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 09:46:11.118762 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20260812 09:46:11.419366 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20260812 09:46:11.780604 5499 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:46:11.780802 5499 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:46:11.781414 5499 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:46:11.782656 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:46:11.783039 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:46:11.783136 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:46:11.783335 5499 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:46:12.369624 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:46:13.125742 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 09:46:13.218358 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 09:46:13.492488 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20260812 09:46:13.906872 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 09:46:15.554569 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 09:46:16.000236 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 09:46:16.062310 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20260812 09:46:16.298451 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 09:46:17.497970 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:46:17.896490 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 09:46:18.461242 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 09:46:18.815685 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 09:46:18.830616 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20260812 09:46:20.726487 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 09:46:20.856105 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260812 09:46:21.257901 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 09:46:21.357059 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20260812 09:46:22.680406 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:46:23.210523 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260812 09:46:23.421056 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260812 09:46:23.804108 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 09:46:23.817965 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20260812 09:46:24.918864 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (180 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:46:25.535933 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 09:46:25.863416 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260812 09:46:26.227494 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20260812 09:46:26.348357 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20260812 09:46:26.565866 5499 consensus_queue.cc:579] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [LEADER]: Leader has been unable to successfully communicate with peer fe00076da7784496bd55b00c1661bf13 for more than 300 seconds (300.109s)
I20260812 09:46:26.813723 5499 consensus_queue.cc:579] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [LEADER]: Leader has been unable to successfully communicate with peer fe00076da7784496bd55b00c1661bf13 for more than 300 seconds (300.501s)
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:46:26.911057 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:46:27.764853 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:46:27.889418 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 09:46:28.439728 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20260812 09:46:28.529997 5506 consensus_queue.cc:579] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 [LEADER]: Leader has been unable to successfully communicate with peer 51bce1a9cf8949ae905ebcb8ddce105b for more than 300 seconds (300.411s)
I20260812 09:46:28.814821 5526 consensus_queue.cc:579] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 [LEADER]: Leader has been unable to successfully communicate with peer 51bce1a9cf8949ae905ebcb8ddce105b for more than 300 seconds (300.422s)
W20260812 09:46:28.816543 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20260812 09:46:28.917315 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20260812 09:46:29.591882 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:46:29.592075 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:46:29.592617 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:46:29.593847 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:46:29.593984 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:46:29.594086 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:46:29.594290 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:46:30.553769 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 09:46:31.059818 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 09:46:31.153883 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 09:46:31.297724 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20260812 09:46:32.806687 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:46:32.917006 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 09:46:33.554286 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 09:46:33.647149 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 09:46:33.778458 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20260812 09:46:35.598887 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 09:46:36.000360 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 09:46:36.024662 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20260812 09:46:36.062034 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 09:46:38.004281 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:46:38.158237 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 09:46:38.565018 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 09:46:38.640664 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20260812 09:46:38.679519 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 09:46:40.662779 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 09:46:41.081418 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 09:46:41.364856 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 09:46:41.461500 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20260812 09:46:43.155718 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:46:43.415501 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 09:46:43.755420 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 09:46:43.766068 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 09:46:44.214277 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20260812 09:46:45.749517 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20260812 09:46:45.772557 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:46:45.772732 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:46:45.773288 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:46:45.774549 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:46:45.774720 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:46:45.774827 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:46:45.775003 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:46:46.144457 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 09:46:46.163261 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 09:46:46.734710 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20260812 09:46:48.188755 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:46:48.388346 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 09:46:48.644907 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 09:46:48.819701 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 09:46:49.161247 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20260812 09:46:50.984517 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 09:46:51.171350 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 09:46:51.324884 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 09:46:51.836315 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20260812 09:46:53.254730 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:46:53.499323 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 09:46:53.656066 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 09:46:53.986542 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 09:46:54.394816 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20260812 09:46:55.979877 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 09:46:56.221185 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 09:46:56.459151 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 09:46:56.862995 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20260812 09:46:58.279215 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:46:58.280210 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 09:46:58.638041 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20260812 09:46:58.927784 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 09:46:59.425383 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20260812 09:46:59.643205 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:46:59.643394 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:46:59.643882 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:46:59.645180 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:46:59.645540 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:46:59.645606 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:46:59.645746 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:00.785822 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 09:47:00.989060 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 09:47:01.607762 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 09:47:01.989001 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20260812 09:47:03.116070 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 09:47:03.335682 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:47:03.466856 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20260812 09:47:04.000494 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 09:47:04.486177 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20260812 09:47:05.267204 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:47:05.267537 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:47:05.268149 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:47:05.269423 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:47:05.269572 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:47:05.269667 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:47:05.269867 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:05.831926 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 09:47:05.908321 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 09:47:06.475080 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 09:47:07.117008 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20260812 09:47:08.356549 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:47:08.357420 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 09:47:08.499454 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 09:47:08.691363 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 09:47:09.701313 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20260812 09:47:11.075052 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 09:47:11.236662 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 09:47:11.330080 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 09:47:12.064412 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20260812 09:47:13.586426 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:47:13.659901 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 09:47:13.706509 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 09:47:13.805954 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 09:47:14.517190 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20260812 09:47:16.262290 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 09:47:16.383939 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 09:47:16.392467 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 09:47:17.192716 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20260812 09:47:18.642704 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:47:18.643541 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 09:47:18.725214 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 09:47:19.066780 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 09:47:19.827253 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20260812 09:47:20.774664 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 09:47:21.223030 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 09:47:21.836433 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 09:47:22.572038 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20260812 09:47:23.159299 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 09:47:23.610136 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 09:47:23.673108 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20260812 09:47:24.220846 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:47:24.221052 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:47:24.221690 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:47:24.223029 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:47:24.223201 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:47:24.223263 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:47:24.223474 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:24.491199 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 09:47:25.188050 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20260812 09:47:25.646865 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 09:47:25.887290 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (243 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:47:26.015272 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 09:47:26.671491 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 09:47:27.894305 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20260812 09:47:28.084816 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:47:28.132041 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:47:28.652618 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 09:47:28.864894 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:47:29.381521 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 09:47:30.368657 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20260812 09:47:30.702179 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 09:47:31.264654 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 09:47:31.706913 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260812 09:47:33.128046 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20260812 09:47:33.370945 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 09:47:33.707604 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20260812 09:47:33.958576 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:47:34.025563 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 09:47:35.409050 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20260812 09:47:36.055670 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:47:36.055999 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:47:36.056492 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:47:36.057852 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:47:36.058018 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:47:36.058087 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:47:36.058225 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:36.072350 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 09:47:36.139550 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 09:47:36.365212 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20260812 09:47:37.853598 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:47:37.853780 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:47:37.854357 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:47:37.855703 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:47:37.855844 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:47:37.855963 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:47:37.856130 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:38.086791 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20260812 09:47:38.516732 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 09:47:38.601908 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 09:47:38.775650 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 09:47:39.018652 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260812 09:47:40.294162 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20260812 09:47:40.988840 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 09:47:41.255736 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 09:47:41.283277 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 09:47:42.487563 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20260812 09:47:43.641733 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 09:47:43.744642 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 09:47:43.815822 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 09:47:44.032564 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:47:44.927685 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20260812 09:47:46.283702 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 09:47:46.318660 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 09:47:46.430217 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 09:47:47.311554 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20260812 09:47:48.500602 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20260812 09:47:48.674911 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:47:48.675092 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:47:48.675788 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:47:48.677037 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:47:48.677270 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:47:48.677395 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:47:48.677553 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:47:48.952987 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 09:47:49.057027 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:47:49.057880 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 09:47:49.738255 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20260812 09:47:51.106215 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 09:47:51.608299 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 09:47:51.653836 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 09:47:52.104739 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20260812 09:47:53.629158 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 09:47:54.106948 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260812 09:47:54.131093 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 09:47:54.231253 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 09:47:54.877970 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20260812 09:47:56.003141 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 09:47:56.672139 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 09:47:56.852151 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 09:47:57.388986 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20260812 09:47:58.548882 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20260812 09:47:59.015738 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 09:47:59.071014 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 09:47:59.426800 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:48:00.015712 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20260812 09:48:00.110944 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:00.111124 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:00.111691 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:00.113049 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:48:00.113149 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:48:00.113214 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:00.113408 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:01.053156 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 09:48:01.531281 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 09:48:01.769579 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 09:48:02.603884 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20260812 09:48:03.452701 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 09:48:04.012391 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 09:48:04.267684 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 09:48:04.466845 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:48:05.088933 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20260812 09:48:05.853888 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 09:48:06.492077 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 09:48:06.714169 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 09:48:07.521474 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20260812 09:48:08.463440 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 09:48:08.841207 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 09:48:09.328680 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 09:48:09.648275 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:48:10.265939 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20260812 09:48:11.223642 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 09:48:11.430512 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 09:48:11.878381 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20260812 09:48:12.513833 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:12.514020 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:12.514590 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:12.516024 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:48:12.516135 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:48:12.516237 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:12.516420 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:12.731156 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20260812 09:48:13.967338 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 09:48:14.029812 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 09:48:14.131767 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 09:48:14.671514 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:48:15.148036 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20260812 09:48:16.163151 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 09:48:16.275910 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20260812 09:48:16.388712 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:16.388890 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:16.389498 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:16.390975 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:48:16.391085 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:48:16.391191 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:16.391372 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:16.562222 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 09:48:17.788625 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20260812 09:48:18.598017 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 09:48:18.676362 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 09:48:19.019593 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 09:48:19.730221 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:48:20.138268 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20260812 09:48:21.194489 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 09:48:21.241037 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 09:48:21.283355 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 09:48:22.744644 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20260812 09:48:23.446560 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 09:48:23.538091 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20260812 09:48:23.803614 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 09:48:24.777526 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:48:25.399135 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20260812 09:48:25.938803 5526 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:25.938993 5526 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:25.939605 5526 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:25.940871 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:48:25.941043 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:48:25.941161 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:25.941381 5526 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:25.962329 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (303 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:48:25.967852 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 09:48:26.086512 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 09:48:26.225612 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 09:48:27.886236 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20260812 09:48:28.575255 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 09:48:28.667876 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 09:48:28.723152 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:48:29.353157 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
I20260812 09:48:29.568881 5572 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:29.569069 5572 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:29.569706 5572 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:29.571151 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:48:29.571259 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:48:29.571324 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:29.571482 5572 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:29.946766 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260812 09:48:30.411814 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20260812 09:48:30.935391 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 09:48:30.935909 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 09:48:31.162552 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 09:48:32.865901 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20260812 09:48:33.534587 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 09:48:33.606258 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 09:48:33.710808 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 09:48:34.970371 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:48:35.101404 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20260812 09:48:35.907579 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 09:48:36.002715 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 09:48:36.053850 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 09:48:37.627983 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20260812 09:48:38.474561 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 09:48:38.532362 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 09:48:38.664630 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 09:48:40.008286 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:48:40.009105 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20260812 09:48:40.839860 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 09:48:40.980329 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 09:48:41.260586 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 09:48:42.552590 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20260812 09:48:43.091408 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 09:48:43.380935 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 09:48:43.539707 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20260812 09:48:44.548979 5572 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:44.549201 5572 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:44.549762 5572 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:44.551256 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:48:44.551374 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:48:44.551515 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:44.551699 5572 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:45.152776 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:48:45.153761 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20260812 09:48:45.781052 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 09:48:45.888562 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 09:48:46.019495 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260812 09:48:47.671517 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20260812 09:48:48.591799 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20260812 09:48:48.630716 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 09:48:48.684520 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 09:48:50.218788 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260812 09:48:50.259183 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20260812 09:48:51.036093 5599 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:48:51.036266 5599 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:48:51.036851 5599 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:48:51.038168 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:48:51.038314 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:48:51.038388 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:48:51.038563 5599 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:48:51.111483 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 09:48:51.211045 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 09:48:51.351897 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 09:48:52.889607 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20260812 09:48:53.571929 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 09:48:53.616240 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 09:48:53.852511 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 09:48:55.265095 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:48:55.371776 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20260812 09:48:56.011111 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 09:48:56.200812 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 09:48:56.474385 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 09:48:57.653187 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20260812 09:48:58.657974 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 09:48:58.993702 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 09:48:59.159865 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20260812 09:48:59.951280 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20260812 09:49:00.005774 5602 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:00.005975 5602 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:00.006458 5602 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:00.007655 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:49:00.007771 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:49:00.007891 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:00.008091 5602 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:00.475061 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:49:01.206542 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 09:49:01.342140 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 09:49:01.614672 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 09:49:02.422819 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20260812 09:49:03.506428 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 09:49:03.696803 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20260812 09:49:03.711397 5602 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:03.711608 5602 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:03.712244 5602 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:03.713562 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:49:03.713776 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:49:03.713872 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:03.714090 5602 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:03.908905 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 09:49:04.940932 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20260812 09:49:05.511567 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:49:06.043982 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 09:49:06.160809 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 09:49:06.518182 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 09:49:07.422344 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20260812 09:49:08.447496 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 09:49:08.481423 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 09:49:09.025137 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20260812 09:49:09.363453 5610 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:09.363627 5610 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:09.364100 5610 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:09.365324 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:49:09.365484 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:49:09.365576 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:09.365753 5610 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:09.835997 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20260812 09:49:10.805632 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260812 09:49:10.911983 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 09:49:10.954687 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 09:49:11.417706 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 09:49:12.384055 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20260812 09:49:13.281522 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 09:49:13.418062 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 09:49:13.929375 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 09:49:14.906158 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20260812 09:49:15.786175 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 09:49:15.836512 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:49:15.837391 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 09:49:16.196866 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 09:49:16.970834 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20260812 09:49:18.132614 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 09:49:18.376533 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 09:49:18.856644 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 09:49:19.542052 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20260812 09:49:20.481595 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 09:49:20.866861 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:49:20.867697 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 09:49:21.255997 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20260812 09:49:21.688258 5614 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:21.688431 5614 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:21.688885 5614 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:21.689909 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:49:21.690064 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:49:21.690152 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:21.690331 5614 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:22.068586 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20260812 09:49:22.863773 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 09:49:23.209641 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 09:49:23.610224 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20260812 09:49:24.166035 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20260812 09:49:24.221655 5614 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:24.221829 5614 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:24.222425 5614 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:24.223629 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:49:24.223737 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:49:24.223860 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:24.224026 5614 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:25.132617 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 09:49:25.645267 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 09:49:25.876101 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20260812 09:49:26.036720 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (363 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:49:26.240475 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 09:49:26.981520 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20260812 09:49:27.428932 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 09:49:28.280589 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 09:49:28.713776 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 09:49:29.315814 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20260812 09:49:30.020987 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20260812 09:49:30.067617 5621 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:30.067816 5621 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:30.068377 5621 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:30.069607 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:49:30.069738 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:49:30.069839 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:30.069983 5621 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:49:30.570827 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:49:30.797916 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 09:49:30.993968 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:49:31.207192 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 09:49:31.968123 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20260812 09:49:32.866222 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 09:49:33.238631 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 09:49:33.706621 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 09:49:34.497786 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20260812 09:49:35.543386 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 09:49:35.754997 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 09:49:36.131138 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:49:36.195283 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 09:49:36.848486 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20260812 09:49:37.978252 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 09:49:38.059954 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 09:49:38.784122 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 09:49:39.213471 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20260812 09:49:40.198873 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 09:49:40.631321 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 09:49:41.023972 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 09:49:41.240900 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:49:42.035742 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20260812 09:49:42.620487 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 09:49:43.192982 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 09:49:43.631156 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 09:49:44.588434 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20260812 09:49:45.136397 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 09:49:45.801362 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 09:49:46.053476 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 09:49:46.431483 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:49:47.041294 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20260812 09:49:47.467464 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 09:49:48.196702 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 09:49:48.622112 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20260812 09:49:49.380188 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20260812 09:49:49.651428 5621 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:49.651677 5621 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:49.652216 5621 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:49.653712 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:49:49.653883 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:49:49.654018 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:49.654225 5621 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:49.866519 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 09:49:50.405305 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 09:49:51.263307 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 09:49:51.472141 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:49:51.945709 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20260812 09:49:52.252084 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 09:49:52.839927 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 09:49:53.707561 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 09:49:54.345605 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20260812 09:49:54.859323 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 09:49:55.088655 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20260812 09:49:56.126065 5621 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:49:56.126256 5621 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:49:56.126811 5621 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:49:56.128108 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:49:56.128218 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:49:56.128305 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:49:56.128480 5621 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:49:56.333968 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 09:49:56.552908 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:49:56.982407 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20260812 09:49:57.259704 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 09:49:57.649058 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 09:49:59.159521 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 09:49:59.559474 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20260812 09:49:59.619990 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 09:50:00.089190 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 09:50:01.575306 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:50:01.978012 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 09:50:01.990464 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 09:50:02.268798 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20260812 09:50:02.450515 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 09:50:04.291565 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 09:50:04.612140 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 09:50:04.714671 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 09:50:04.813663 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20260812 09:50:06.426998 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 09:50:06.584374 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:50:06.974238 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 09:50:07.051772 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 09:50:07.346922 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20260812 09:50:08.877458 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 09:50:09.358191 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 09:50:09.553926 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 09:50:09.849426 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20260812 09:50:11.450954 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 09:50:11.658464 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:50:11.720722 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 09:50:12.094985 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 09:50:12.614560 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20260812 09:50:13.812384 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20260812 09:50:14.151258 5621 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:50:14.151460 5621 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:50:14.152076 5621 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:50:14.153488 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:50:14.153600 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:50:14.153672 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:50:14.153853 5621 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:50:14.224606 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 09:50:14.351845 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 09:50:15.064572 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20260812 09:50:16.449048 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 09:50:16.715224 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:50:16.716161 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 09:50:16.867885 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 09:50:17.670243 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20260812 09:50:18.921300 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 09:50:19.147190 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 09:50:19.562781 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 09:50:20.139060 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20260812 09:50:21.461324 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 09:50:21.842440 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:50:21.843317 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 09:50:21.882098 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 09:50:22.666607 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20260812 09:50:23.918427 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 09:50:24.353626 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 09:50:24.592667 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 09:50:25.148412 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20260812 09:50:26.108865 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (423 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:50:26.124375 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 09:50:26.871480 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:50:26.872282 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 09:50:26.912040 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 09:50:27.512235 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20260812 09:50:28.615800 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20260812 09:50:29.415105 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 09:50:29.648914 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 09:50:30.254321 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20260812 09:50:30.895259 5621 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:50:30.895437 5621 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:50:30.896037 5621 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:50:30.897262 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:50:30.897396 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:50:30.897492 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:50:30.897677 5621 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:50:31.510529 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 09:50:31.513183 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:50:31.714054 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:50:31.971190 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:50:31.972113 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 09:50:32.762810 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20260812 09:50:33.881762 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 09:50:34.358953 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 09:50:34.473721 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 09:50:35.140416 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20260812 09:50:36.336724 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 09:50:36.942833 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 09:50:37.045817 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:50:37.129329 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 09:50:37.450465 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20260812 09:50:38.526981 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 09:50:39.610816 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 09:50:39.903893 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 09:50:39.946156 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20260812 09:50:40.893496 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 09:50:42.141855 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:50:42.142664 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 09:50:42.189002 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20260812 09:50:42.401408 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 09:50:43.384519 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 09:50:44.656486 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 09:50:44.837795 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20260812 09:50:44.928597 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 09:50:45.714785 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 09:50:46.822758 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 09:50:47.271701 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:50:47.272475 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 09:50:47.313886 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20260812 09:50:47.852033 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 09:50:49.332978 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20260812 09:50:49.437295 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20260812 09:50:49.454939 5665 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:50:49.455123 5665 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:50:49.455691 5665 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:50:49.456840 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:50:49.456988 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:50:49.457090 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:50:49.457284 5665 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:50:49.519948 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 09:50:50.466099 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 09:50:51.501482 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 09:50:51.900846 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20260812 09:50:51.909191 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 09:50:52.290839 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20260812 09:50:52.684679 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 09:50:53.912137 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 09:50:54.255368 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 09:50:54.447659 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20260812 09:50:54.970466 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 09:50:56.541692 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 09:50:56.669503 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20260812 09:50:56.799726 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 09:50:57.237365 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 09:50:57.322860 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:50:59.013702 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 09:50:59.027782 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20260812 09:50:59.366565 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 09:50:59.733913 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 09:51:01.377218 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 09:51:01.552834 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20260812 09:51:01.712280 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 09:51:02.100558 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 09:51:02.475415 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20260812 09:51:02.522085 5665 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:02.522264 5665 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:02.522761 5665 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:02.523936 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:02.524107 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:02.524221 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:02.524394 5665 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:03.615013 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 09:51:04.146499 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20260812 09:51:04.247601 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 09:51:04.632344 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 09:51:06.069862 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 09:51:06.600031 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 09:51:06.641232 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20260812 09:51:07.182583 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 09:51:07.664880 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:51:08.952129 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 09:51:09.146590 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20260812 09:51:09.481923 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 09:51:09.806326 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 09:51:11.363215 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 09:51:11.525652 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20260812 09:51:11.787587 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 09:51:12.666163 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:51:12.666872 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 09:51:13.715111 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 09:51:13.821449 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20260812 09:51:14.485481 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 09:51:15.320617 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 09:51:16.196569 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 09:51:16.287689 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20260812 09:51:17.086066 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 09:51:17.716173 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:51:17.716882 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20260812 09:51:17.838215 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:17.838387 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:17.838867 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:17.840034 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:51:17.840233 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:51:17.840291 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:17.840430 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:18.866873 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 09:51:19.072310 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20260812 09:51:19.532243 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 09:51:20.411309 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260812 09:51:21.211792 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 09:51:21.425590 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20260812 09:51:22.081017 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 09:51:22.752899 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:51:23.063534 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 09:51:23.731832 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 09:51:23.852718 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20260812 09:51:24.701458 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260812 09:51:25.551440 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 09:51:26.182701 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (483 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:51:26.199488 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 09:51:26.536913 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20260812 09:51:27.150561 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 09:51:27.767504 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:51:28.235845 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 09:51:28.628422 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20260812 09:51:29.021576 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:29.021755 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:29.022322 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:29.023557 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:29.023708 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:29.023821 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:29.023995 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:29.117375 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
W20260812 09:51:29.574478 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 09:51:30.869736 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 09:51:30.902143 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 09:51:31.423290 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20260812 09:51:32.154042 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 09:51:32.925005 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 44 similar messages]
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:51:32.935626 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:51:33.213354 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 09:51:33.543735 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 09:51:33.971798 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1211: this message will repeat every 5th retry.
W20260812 09:51:34.794755 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 09:51:35.522421 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 09:51:35.998042 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 09:51:36.466199 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1216: this message will repeat every 5th retry.
W20260812 09:51:37.567420 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 09:51:37.960968 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20260812 09:51:38.071055 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 09:51:38.538801 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
W20260812 09:51:38.782224 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1221: this message will repeat every 5th retry.
I20260812 09:51:40.192554 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:40.192728 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:40.193248 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:40.193845 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 09:51:40.194320 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:40.194518 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:40.194599 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:40.194729 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:40.404577 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 09:51:41.146679 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 09:51:41.422967 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1226: this message will repeat every 5th retry.
W20260812 09:51:42.443570 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 09:51:42.986337 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:51:42.987205 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 09:51:43.751785 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 09:51:44.015364 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1231: this message will repeat every 5th retry.
W20260812 09:51:44.757822 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
I20260812 09:51:44.774979 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:44.775172 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:44.775583 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:44.776530 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:44.776647 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:44.776710 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:44.776851 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:45.674670 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 09:51:46.108435 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 09:51:46.781085 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1236: this message will repeat every 5th retry.
W20260812 09:51:47.197923 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 09:51:48.075610 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:51:48.195792 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 09:51:48.678840 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 09:51:49.277138 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1241: this message will repeat every 5th retry.
W20260812 09:51:49.874580 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 09:51:50.672236 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 09:51:51.034970 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
I20260812 09:51:51.359407 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:51.359704 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:51.360317 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:51.361786 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:51.361940 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:51.362011 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:51.362190 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:51.680756 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1246: this message will repeat every 5th retry.
W20260812 09:51:52.529187 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 09:51:52.821199 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 09:51:53.113155 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:51:53.142429 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 09:51:54.179260 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1251: this message will repeat every 5th retry.
W20260812 09:51:54.987206 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 09:51:55.349303 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 09:51:55.464284 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
W20260812 09:51:56.662078 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1256: this message will repeat every 5th retry.
I20260812 09:51:56.793984 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:51:56.794175 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:51:56.794822 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:51:56.796167 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:51:56.796254 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:51:56.796303 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:51:56.796478 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:51:57.589830 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 09:51:57.639485 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 09:51:57.964367 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 09:51:58.200729 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:51:59.297714 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1261: this message will repeat every 5th retry.
W20260812 09:52:00.069401 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 09:52:00.343179 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 09:52:00.397822 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 09:52:01.760540 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1266: this message will repeat every 5th retry.
W20260812 09:52:02.343658 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 09:52:02.945278 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 09:52:03.006654 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 09:52:03.241068 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:04.312175 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1271: this message will repeat every 5th retry.
W20260812 09:52:04.700479 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 09:52:05.297081 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 09:52:05.722430 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
I20260812 09:52:05.936849 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:52:05.937084 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:52:05.937669 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:52:05.938990 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:52:05.939116 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:52:05.939224 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:52:05.939429 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:52:06.668939 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1276: this message will repeat every 5th retry.
W20260812 09:52:07.311522 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 09:52:07.747149 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 09:52:08.242316 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:08.263228 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 09:52:09.061798 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1281: this message will repeat every 5th retry.
W20260812 09:52:09.865231 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 09:52:10.175220 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 09:52:10.732033 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 09:52:11.725589 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1286: this message will repeat every 5th retry.
W20260812 09:52:12.353472 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 09:52:12.692368 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 09:52:13.245959 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:52:13.246695 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 09:52:14.647472 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1291: this message will repeat every 5th retry.
W20260812 09:52:14.937760 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 09:52:14.962916 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 09:52:15.346794 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 09:52:17.028115 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1296: this message will repeat every 5th retry.
W20260812 09:52:17.557113 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 09:52:17.597206 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 09:52:17.983232 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 09:52:18.397131 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:19.630687 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1301: this message will repeat every 5th retry.
W20260812 09:52:19.889789 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 09:52:19.929636 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 09:52:20.549700 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 09:52:22.161574 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1306: this message will repeat every 5th retry.
W20260812 09:52:22.238411 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 09:52:22.366269 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 09:52:22.800264 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 09:52:23.474676 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20260812 09:52:23.923717 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:52:23.923960 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:52:23.924582 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:52:23.925922 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:52:23.926064 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:52:23.926183 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:52:23.926363 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:52:24.387969 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 09:52:24.531584 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1311: this message will repeat every 5th retry.
W20260812 09:52:24.862962 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 09:52:25.060628 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 09:52:26.255712 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (543 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
I20260812 09:52:26.366721 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:52:26.366896 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:52:26.367460 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:52:26.368654 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:52:26.368767 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:52:26.368853 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:52:26.369033 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:52:26.718561 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 09:52:27.195701 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1316: this message will repeat every 5th retry.
W20260812 09:52:27.633463 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 09:52:27.734282 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 09:52:28.567065 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20260812 09:52:29.388792 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 09:52:29.812121 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1321: this message will repeat every 5th retry.
W20260812 09:52:29.922716 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 09:52:30.249739 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 09:52:31.535674 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 09:52:32.396785 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 09:52:32.426537 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1326: this message will repeat every 5th retry.
W20260812 09:52:32.783367 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 09:52:33.586822 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:34.087775 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:52:34.143953 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:52:34.760340 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 09:52:35.002969 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 09:52:35.194661 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1331: this message will repeat every 5th retry.
W20260812 09:52:36.722450 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 09:52:37.431484 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 09:52:37.464025 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 09:52:37.924924 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1336: this message will repeat every 5th retry.
W20260812 09:52:38.590911 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:52:39.337321 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 09:52:39.657188 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 09:52:40.069532 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 09:52:40.585976 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1341: this message will repeat every 5th retry.
W20260812 09:52:41.696239 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 09:52:41.966734 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
I20260812 09:52:42.153259 5672 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:52:42.153473 5672 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:52:42.154012 5672 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:52:42.155355 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:52:42.155457 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:52:42.155570 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:52:42.155754 5672 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:52:42.446108 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 09:52:43.278337 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1346: this message will repeat every 5th retry.
W20260812 09:52:43.670630 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:52:44.227295 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 09:52:44.229907 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 09:52:44.761770 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 09:52:45.400110 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1351: this message will repeat every 5th retry.
W20260812 09:52:46.770275 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 09:52:46.859570 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 09:52:47.294916 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 09:52:47.820863 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1356: this message will repeat every 5th retry.
W20260812 09:52:48.801426 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:49.496316 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 09:52:49.608990 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 09:52:49.831020 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 09:52:50.359373 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1361: this message will repeat every 5th retry.
W20260812 09:52:51.882462 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 09:52:51.976140 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 09:52:52.284525 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 09:52:52.869840 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1366: this message will repeat every 5th retry.
W20260812 09:52:53.909993 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:54.511811 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 09:52:54.546236 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 09:52:54.724140 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 09:52:55.609510 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1371: this message will repeat every 5th retry.
W20260812 09:52:56.791923 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 09:52:57.083199 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 09:52:57.265082 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 09:52:58.285578 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1376: this message will repeat every 5th retry.
W20260812 09:52:59.187134 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:52:59.238941 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
I20260812 09:52:59.401129 5738 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:52:59.401319 5738 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:52:59.401984 5738 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:52:59.403226 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:52:59.403326 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:52:59.403390 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:52:59.403578 5738 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:52:59.585551 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 09:52:59.759151 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 09:53:00.696254 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1381: this message will repeat every 5th retry.
W20260812 09:53:01.918177 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 09:53:02.097563 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 09:53:02.152256 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 09:53:03.353236 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1386: this message will repeat every 5th retry.
W20260812 09:53:04.344867 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:53:04.491367 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 09:53:04.587975 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 09:53:05.053093 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 09:53:05.830324 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1391: this message will repeat every 5th retry.
W20260812 09:53:07.044809 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 09:53:07.195818 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 09:53:07.531889 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 09:53:08.635445 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1396: this message will repeat every 5th retry.
W20260812 09:53:09.468575 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:53:09.581624 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
W20260812 09:53:09.681221 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 09:53:10.027266 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 09:53:10.788082 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1401: this message will repeat every 5th retry.
W20260812 09:53:12.172164 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 09:53:12.239547 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 09:53:12.716946 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 09:53:13.238286 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1406: this message will repeat every 5th retry.
I20260812 09:53:14.270262 5742 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:53:14.270468 5742 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:53:14.270990 5742 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:53:14.272378 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:53:14.272490 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:53:14.272554 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:53:14.272712 5742 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:53:14.541989 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:53:14.543010 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 09:53:14.658747 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 09:53:15.200217 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 09:53:16.175263 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1411: this message will repeat every 5th retry.
W20260812 09:53:17.211048 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 09:53:17.257675 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 09:53:17.587781 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 09:53:18.598124 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1416: this message will repeat every 5th retry.
W20260812 09:53:19.554325 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:53:19.742638 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 09:53:19.783263 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 09:53:20.038676 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 09:53:20.985428 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1421: this message will repeat every 5th retry.
W20260812 09:53:22.255940 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 09:53:22.286957 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 09:53:22.604549 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 09:53:23.696622 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1426: this message will repeat every 5th retry.
W20260812 09:53:24.751390 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:53:24.812543 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 09:53:24.853083 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 09:53:25.325979 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 09:53:26.187582 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1431: this message will repeat every 5th retry.
W20260812 09:53:26.326731 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (603 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:53:26.909179 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 09:53:27.400712 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 09:53:28.040709 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 09:53:28.495350 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1436: this message will repeat every 5th retry.
W20260812 09:53:29.427105 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
W20260812 09:53:29.863788 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:53:29.864562 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 09:53:30.713513 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 09:53:30.998267 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1441: this message will repeat every 5th retry.
W20260812 09:53:32.049094 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 09:53:32.431764 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 09:53:33.179561 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 09:53:33.368111 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1446: this message will repeat every 5th retry.
I20260812 09:53:34.164430 5742 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:53:34.164602 5742 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:53:34.165172 5742 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:53:34.166550 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:53:34.166661 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:53:34.166721 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:53:34.166940 5742 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:53:34.514477 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 09:53:34.857003 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 09:53:35.021817 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 43 similar messages]
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:53:35.343638 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:53:35.453037 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 09:53:35.736635 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1451: this message will repeat every 5th retry.
W20260812 09:53:36.748096 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 09:53:37.358518 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 09:53:37.869220 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1456: this message will repeat every 5th retry.
W20260812 09:53:38.088121 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 09:53:39.130354 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 09:53:39.965788 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 09:53:40.049556 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:53:40.326141 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1461: this message will repeat every 5th retry.
W20260812 09:53:40.975044 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 09:53:41.565060 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 09:53:42.467741 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 09:53:42.898995 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1466: this message will repeat every 5th retry.
W20260812 09:53:43.500298 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 09:53:44.354092 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 09:53:45.054909 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:53:45.055687 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 09:53:45.580461 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1471: this message will repeat every 5th retry.
W20260812 09:53:45.695519 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 09:53:46.839138 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 09:53:47.817330 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 09:53:48.210227 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1476: this message will repeat every 5th retry.
W20260812 09:53:48.304849 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 09:53:49.501410 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
I20260812 09:53:49.552093 5742 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:53:49.552284 5742 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:53:49.552873 5742 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:53:49.554222 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:53:49.554358 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:53:49.554472 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:53:49.554644 5742 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:53:50.087415 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:53:50.419425 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 09:53:50.605603 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1481: this message will repeat every 5th retry.
W20260812 09:53:50.882383 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 09:53:52.042411 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 09:53:52.967123 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 09:53:53.145572 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1486: this message will repeat every 5th retry.
W20260812 09:53:53.608089 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 09:53:54.799768 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 09:53:55.210335 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20260812 09:53:55.654628 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 09:53:55.880568 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1491: this message will repeat every 5th retry.
W20260812 09:53:56.234601 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 09:53:57.007798 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 09:53:58.129081 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 09:53:58.445245 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1496: this message will repeat every 5th retry.
W20260812 09:53:58.754868 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
I20260812 09:53:59.395939 5742 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:53:59.396173 5742 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:53:59.396687 5742 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:53:59.398020 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:53:59.398185 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:53:59.398249 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:53:59.398401 5742 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:53:59.641825 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 09:54:00.275622 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:54:00.888291 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 09:54:01.069139 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1501: this message will repeat every 5th retry.
W20260812 09:54:01.314139 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 09:54:02.143981 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 09:54:03.431777 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1506: this message will repeat every 5th retry.
W20260812 09:54:03.633942 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 09:54:03.780718 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 09:54:04.487835 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 09:54:05.328639 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:54:05.722088 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1511: this message will repeat every 5th retry.
W20260812 09:54:06.111064 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 09:54:06.530367 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 09:54:07.205755 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 09:54:08.122807 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1516: this message will repeat every 5th retry.
W20260812 09:54:08.608170 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 09:54:09.088001 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 09:54:09.675361 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 09:54:10.351517 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:54:10.692517 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1521: this message will repeat every 5th retry.
W20260812 09:54:10.896904 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 09:54:11.740605 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 09:54:12.213743 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 09:54:13.251202 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1526: this message will repeat every 5th retry.
W20260812 09:54:13.345275 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
I20260812 09:54:14.154022 5775 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:14.154225 5775 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:14.154781 5775 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:14.155956 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:14.156140 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:14.156226 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:14.156387 5775 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:14.463815 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 09:54:14.575562 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 09:54:15.401306 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:54:15.455935 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1531: this message will repeat every 5th retry.
W20260812 09:54:15.945956 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 09:54:17.085471 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 09:54:17.109890 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 09:54:18.014140 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1536: this message will repeat every 5th retry.
W20260812 09:54:18.164583 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 09:54:19.636744 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 09:54:19.813704 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 09:54:20.434636 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:54:20.493400 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1541: this message will repeat every 5th retry.
W20260812 09:54:20.829700 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 09:54:22.326097 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 09:54:22.380384 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 09:54:22.899194 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1546: this message will repeat every 5th retry.
W20260812 09:54:23.378053 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 09:54:24.849197 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 09:54:25.149632 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 09:54:25.491079 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20260812 09:54:25.491881 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1551: this message will repeat every 5th retry.
W20260812 09:54:25.783581 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 09:54:26.400143 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (663 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:54:27.213522 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 09:54:27.724236 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 09:54:27.851114 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1556: this message will repeat every 5th retry.
W20260812 09:54:28.335420 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 09:54:29.951236 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 09:54:30.059335 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1561: this message will repeat every 5th retry.
W20260812 09:54:30.288194 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 09:54:30.573127 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:54:30.921308 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 09:54:32.416406 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 09:54:32.740434 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 09:54:32.817729 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1566: this message will repeat every 5th retry.
W20260812 09:54:33.464763 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
I20260812 09:54:34.068586 5775 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:34.068761 5775 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:34.069315 5775 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:34.070528 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:34.070827 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:34.070897 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:34.071064 5775 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:35.049553 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 09:54:35.155179 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1571: this message will repeat every 5th retry.
W20260812 09:54:35.560118 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 09:54:35.696173 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:54:36.071751 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:54:36.576942 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
I20260812 09:54:36.716007 5775 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:36.716193 5775 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:36.716775 5775 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:36.718024 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:36.718183 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:36.718266 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:36.718441 5775 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:37.357197 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 09:54:37.826758 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1576: this message will repeat every 5th retry.
W20260812 09:54:37.963220 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 09:54:38.694752 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
I20260812 09:54:38.730440 5775 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:38.730675 5775 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:38.731204 5775 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:38.732407 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:38.732545 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:38.732633 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:38.732813 5775 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:39.790561 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 09:54:40.505784 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1581: this message will repeat every 5th retry.
W20260812 09:54:40.532087 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 09:54:40.768749 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:54:41.420717 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 09:54:42.230706 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 09:54:42.874387 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 09:54:42.998325 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1586: this message will repeat every 5th retry.
W20260812 09:54:43.691342 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
I20260812 09:54:44.193967 5775 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:44.194150 5775 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:44.194666 5775 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:44.195917 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:44.196046 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:44.196158 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:44.196336 5775 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:44.899118 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 09:54:45.321179 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1591: this message will repeat every 5th retry.
W20260812 09:54:45.717761 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 09:54:45.785054 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:54:46.281482 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 09:54:47.573839 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 09:54:47.727649 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1596: this message will repeat every 5th retry.
W20260812 09:54:47.989158 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 09:54:48.948406 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 09:54:49.930469 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 09:54:50.104849 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1601: this message will repeat every 5th retry.
W20260812 09:54:50.740608 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 09:54:51.105583 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:54:51.536769 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 09:54:52.331312 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 09:54:52.621881 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1606: this message will repeat every 5th retry.
W20260812 09:54:53.067607 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 09:54:53.994886 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 09:54:54.867724 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 09:54:55.386355 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1611: this message will repeat every 5th retry.
W20260812 09:54:55.571844 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 09:54:56.180913 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:54:56.529973 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 09:54:57.508456 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 09:54:57.949050 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1616: this message will repeat every 5th retry.
W20260812 09:54:58.155190 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 09:54:58.971218 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
I20260812 09:54:59.806824 5799 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:54:59.807024 5799 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:54:59.807679 5799 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:54:59.808883 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:54:59.809043 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:54:59.809127 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:54:59.809300 5799 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:54:59.849826 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 09:55:00.357203 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1621: this message will repeat every 5th retry.
W20260812 09:55:00.766075 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 09:55:01.197182 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:55:01.587678 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 09:55:02.535342 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 09:55:02.884332 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1626: this message will repeat every 5th retry.
W20260812 09:55:03.254531 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 09:55:03.952584 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 09:55:05.126760 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 09:55:05.442303 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1631: this message will repeat every 5th retry.
W20260812 09:55:05.508221 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
W20260812 09:55:06.413846 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:55:06.418118 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 09:55:07.811141 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 09:55:07.856698 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 09:55:08.251804 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1636: this message will repeat every 5th retry.
I20260812 09:55:08.509187 5799 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:55:08.509505 5799 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:55:08.510151 5799 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:55:08.511518 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:55:08.511677 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:55:08.511765 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:55:08.511945 5799 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:55:09.101094 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 09:55:10.203646 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 09:55:10.375653 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 09:55:10.714005 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1641: this message will repeat every 5th retry.
W20260812 09:55:11.464598 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:55:11.465436 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 09:55:12.569439 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 09:55:12.962811 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 09:55:13.270263 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1646: this message will repeat every 5th retry.
W20260812 09:55:13.882421 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 09:55:15.241303 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 09:55:15.679708 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 09:55:15.750670 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1651: this message will repeat every 5th retry.
W20260812 09:55:16.267884 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 09:55:16.629614 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:55:17.823653 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 09:55:18.104800 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 09:55:18.334388 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1656: this message will repeat every 5th retry.
W20260812 09:55:18.648123 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
I20260812 09:55:19.719687 5815 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:55:19.719871 5815 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:55:19.720398 5815 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:55:19.721544 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:55:19.721673 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:55:19.721776 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:55:19.721918 5815 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:55:20.260046 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 09:55:20.526615 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 09:55:20.698563 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1661: this message will repeat every 5th retry.
W20260812 09:55:21.321225 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 09:55:21.672694 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:55:22.825837 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 09:55:22.939853 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1666: this message will repeat every 5th retry.
W20260812 09:55:23.024391 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 09:55:23.889744 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 09:55:25.292917 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 09:55:25.428323 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1671: this message will repeat every 5th retry.
W20260812 09:55:25.444823 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 09:55:26.242623 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 09:55:26.473308 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (723 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:55:26.875088 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:55:27.615636 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 09:55:27.858024 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1676: this message will repeat every 5th retry.
W20260812 09:55:27.972683 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 09:55:28.620715 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 09:55:30.094751 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1681: this message will repeat every 5th retry.
W20260812 09:55:30.271515 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 09:55:30.422526 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 09:55:30.956039 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 09:55:31.903973 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20260812 09:55:32.336895 5817 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:55:32.337116 5817 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:55:32.337709 5817 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:55:32.339134 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:55:32.339335 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:55:32.339463 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:55:32.339640 5817 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:55:32.491262 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 09:55:32.541988 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1686: this message will repeat every 5th retry.
W20260812 09:55:32.850339 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 09:55:33.447039 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 09:55:34.923557 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1691: this message will repeat every 5th retry.
W20260812 09:55:35.100286 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 09:55:35.476804 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 09:55:35.788153 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 09:55:36.943199 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:55:37.403728 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1696: this message will repeat every 5th retry.
W20260812 09:55:37.567875 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
Master Summary
UUID | Address | Status
--------------------------------+--------------------+-------------
<unknown> (127.31.76.62:44395) | 127.31.76.62:44395 | UNAVAILABLE
Error from 127.31.76.62:44395: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111) (UNAVAILABLE)
All reported replicas are:
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------+--------------+--------------+------------
Tablet Server Summary
Version Summary
Version | Servers
---------+---------
The cluster doesn't have any matching tablets
The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
==================
Warnings:
==================
master unusual flags check error: 1 of 1 masters were not available to retrieve unusual flags
master diverged flags check error: 1 of 1 masters were not available to retrieve time_source category flags
==================
Errors:
==================
Network error: error fetching info from masters: failed to gather info from all masters: 1 of 1 had errors
Not found: master consensus error: no master consensus state available
Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
FAILED
I20260812 09:55:37.795399 32048 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Timed out: leader master liveness check error: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.76.62:44395, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
W20260812 09:55:37.917229 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 09:55:38.124485 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 09:55:39.911032 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1701: this message will repeat every 5th retry.
W20260812 09:55:39.937147 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 09:55:40.474397 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 09:55:40.499845 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 09:55:41.984687 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260812 09:55:42.373479 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1706: this message will repeat every 5th retry.
W20260812 09:55:42.417789 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 09:55:42.861027 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 09:55:42.896818 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 09:55:44.772588 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1711: this message will repeat every 5th retry.
W20260812 09:55:45.022648 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 09:55:45.035682 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 09:55:45.269881 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 09:55:46.979903 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1716: this message will repeat every 5th retry.
W20260812 09:55:46.991452 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:55:47.255488 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 09:55:47.612564 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 09:55:47.844426 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 09:55:49.218518 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1721: this message will repeat every 5th retry.
W20260812 09:55:49.699265 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 09:55:49.955670 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 09:55:50.355273 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
I20260812 09:55:51.028147 5839 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:55:51.028301 5839 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:55:51.028802 5839 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:55:51.030129 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:55:51.030241 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:55:51.030311 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:55:51.030534 5839 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
************************ BEGIN STACKS **************************
[New LWP 32051]
[New LWP 32052]
[New LWP 413]
[New LWP 1116]
[New LWP 1118]
[New LWP 1119]
[New LWP 1121]
[New LWP 5824]
[New LWP 5826]
[New LWP 5827]
[New LWP 5829]
[New LWP 5832]
[New LWP 5834]
[New LWP 5835]
[New LWP 5836]
[New LWP 5837]
W20260812 09:55:51.676136 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1726: this message will repeat every 5th retry.
W20260812 09:55:52.073272 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:55:52.074164 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 09:55:52.544493 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 09:55:53.027079 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 09:55:54.307292 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1731: this message will repeat every 5th retry.
W20260812 09:55:54.452665 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 09:55:55.048063 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 09:55:55.329460 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 09:55:56.601974 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1736: this message will repeat every 5th retry.
W20260812 09:55:56.897136 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 09:55:57.105789 764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:55:57.467402 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 09:55:57.618351 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 09:55:59.054594 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1741: this message will repeat every 5th retry.
W20260812 09:55:59.452625 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 09:55:59.894898 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 09:56:00.131029 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 09:56:01.662263 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1746: this message will repeat every 5th retry.
W20260812 09:56:02.130215 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:56:02.131042 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 09:56:02.403259 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 09:56:02.507694 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
W20260812 09:56:03.997237 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1751: this message will repeat every 5th retry.
I20260812 09:56:04.516817 5839 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:56:04.517051 5839 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:56:04.517643 5839 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:56:04.519070 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
W20260812 09:56:04.519282 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
I20260812 09:56:04.519381 760 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:56:04.519542 5839 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260812 09:56:04.780119 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 09:56:04.974243 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 09:56:05.038219 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 09:56:06.370240 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1756: this message will repeat every 5th retry.
W20260812 09:56:07.130514 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:56:07.131336 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260812 09:56:07.229100 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 09:56:07.395749 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 09:56:09.297879 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1761: this message will repeat every 5th retry.
W20260812 09:56:09.656697 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 09:56:09.773774 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260812 09:56:09.783872 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260812 09:56:12.091786 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1766: this message will repeat every 5th retry.
W20260812 09:56:12.407795 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20260812 09:56:12.408672 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260812 09:56:12.490093 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260812 09:56:12.526501 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260812 09:56:14.636013 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260812 09:56:14.776501 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1771: this message will repeat every 5th retry.
W20260812 09:56:14.950464 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
W20260812 09:56:15.280462 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260812 09:56:17.140774 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1776: this message will repeat every 5th retry.
W20260812 09:56:17.293901 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
W20260812 09:56:17.309929 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
W20260812 09:56:17.607712 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20260812 09:56:18.174490 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260812 09:56:19.586225 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260812 09:56:19.646914 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
0x00007f6d63ed1ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f6d5e51f240 (LWP 32048) "all_types-itest" 0x00007f6d63ed1ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f6d5dcc1700 (LWP 32051) "all_types-itest" 0x00007f6d63ed632a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f6d5c6f8700 (LWP 32052) "kernel-watcher-" 0x00007f6d63ed1fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f6d5acee700 (LWP 413) "rpc reactor-413" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f6d59cec700 (LWP 1116) "rpc reactor-111" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f6d5a4ed700 (LWP 1118) "rpc reactor-111" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f6d5b6f6700 (LWP 1119) "rpc reactor-111" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f6d5bef7700 (LWP 1121) "rpc reactor-112" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f6d51cdc700 (LWP 5824) "rpc reactor-582" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f6d58cea700 (LWP 5826) "rpc reactor-582" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f6d564e5700 (LWP 5827) "rpc reactor-582" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
W20260812 09:56:19.794361 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1781: this message will repeat every 5th retry.
12 Thread 0x7f6d514db700 (LWP 5829) "rpc reactor-582" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f6d57ce8700 (LWP 5832) "rpc reactor-583" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f6d574e7700 (LWP 5834) "rpc reactor-583" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
15 Thread 0x7f6d56ce6700 (LWP 5835) "rpc reactor-583" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
16 Thread 0x7f6d55ce4700 (LWP 5836) "rpc reactor-583" 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
17 Thread 0x7f6d554e3700 (LWP 5837) "client-negotiat" 0x00007f6d63ed1fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
Thread 17 (Thread 0x7f6d554e3700 (LWP 5837)):
#0 0x00007f6d63ed1fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f6d62c76de9 in kudu::ConditionVariable::WaitFor (this=0x7f6d554e2140, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:121
#2 0x00007f6d62dabecf in kudu::ThreadPool::DispatchThread (this=0x3ac3f900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:721
#3 0x00007f6d62dacc83 in kudu::ThreadPool::<lambda()>::operator()(void) const (__closure=0x3b348028) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849
W20260812 09:56:20.557896 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#4 0x00007f6d62dae3f2 in std::_Function_handler<void(), kudu::ThreadPool::CreateThread()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#5 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3b348028) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#6 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3b348000) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#7 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7f6d55ce4700 (LWP 5836)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaec480, timeout=0.099738177144899964) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaec480, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7bd88, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7bd80) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9aef98) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9aef98) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9aef70) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7f6d56ce6700 (LWP 5835)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaef600, timeout=0.099722730927169323) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaef600, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7b3c8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7b3c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9aedf8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9aedf8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9aedd0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f6d574e7700 (LWP 5834)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaedb00, timeout=0.099713892210274935) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaedb00, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7a048, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7a040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9af618) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9af618) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9af5f0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f6d57ce8700 (LWP 5832)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaed200, timeout=0.099189103115350008) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaed200, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aafcee8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aafcee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9afa28) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9afa28) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9afa00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f6d514db700 (LWP 5829)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaec900, timeout=0.099629770033061504) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaec900, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aafd3c8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aafd3c0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3aaad068) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3aaad068) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3aaad040) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f6d564e5700 (LWP 5827)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab45f80, timeout=0.099482514895498753) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab45f80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aafd8a8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aafd8a0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3aaadc98) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3aaadc98) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3aaadc70) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f6d58cea700 (LWP 5826)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab46d00, timeout=0.099440303165465593) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab46d00, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aafd568, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aafd560) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3aaacf98) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3aaacf98) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3aaacf70) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f6d51cdc700 (LWP 5824)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab46400, timeout=0.09940471313893795) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab46400, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aafcd48, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aafcd40) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3aaacc58) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3aaacc58) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3aaacc30) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f6d5bef7700 (LWP 1121)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaecd80, timeout=0.099963792832568288) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaecd80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7a388, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7a380) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9aff08) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9aff08) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9afee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f6d5b6f6700 (LWP 1119)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3aaed680, timeout=0.099535247776657343) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3aaed680, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7aa08, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7aa00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9af2d8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9af2d8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9af2b0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f6d5a4ed700 (LWP 1118)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab47a80, timeout=0.099935431964695454) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab47a80, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7ad48, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7ad40) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9aed28) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9aed28) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9aed00) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f6d59cec700 (LWP 1116)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab47600, timeout=0.099833317333832383) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab47600, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7a868, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7a860) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3a9af478) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9af478) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9af450) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f6d5acee700 (LWP 413)):
#0 0x00007f6d5f2f0947 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6d61d2bbda in epoll_poll (loop=0x3ab47180, timeout=0.099360250867903233) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev_epoll.c:153
#2 0x00007f6d61d2f228 in ev_run (loop=0x3ab47180, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3521
#3 0x00007f6d66fe9207 in ev::loop_ref::run (this=0x3aa7aee8, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f6d66fe434a in kudu::rpc::ReactorThread::RunThread (this=0x3aa7aee0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510
#5 0x00007f6d66fe0fb5 in kudu::rpc::ReactorThread::<lambda()>::operator()(void) const (__closure=0x3aaac778) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197
#6 0x00007f6d66fe844d in std::_Function_handler<void(), kudu::rpc::ReactorThread::Init()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3aaac778) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3aaac750) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f6d5c6f8700 (LWP 32052)):
#0 0x00007f6d63ed1fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f6d62c76c38 in kudu::ConditionVariable::WaitUntil (this=0x3a95f878, until=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:87
#2 0x00007f6d6b82e4e1 in kudu::CountDownLatch::WaitUntil (this=0x3a95f840, when=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:89
#3 0x00007f6d6b82e569 in kudu::CountDownLatch::WaitFor (this=0x3a95f840, delta=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:99
#4 0x00007f6d62d02f57 in kudu::KernelStackWatchdog::RunThread (this=0x3a95f7a0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f6d62d02291 in kudu::KernelStackWatchdog::<lambda()>::operator()(void) const (__closure=0x3a9aeec8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f6d62d037f3 in std::_Function_handler<void(), kudu::KernelStackWatchdog::KernelStackWatchdog()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:297
#7 0x00007f6d6bbe5170 in std::function<void ()>::operator()() const (this=0x3a9aeec8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687
#8 0x00007f6d62d9d512 in kudu::Thread::SuperviseThread (arg=0x3a9aeea0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694
#9 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
W20260812 09:56:21.909755 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
Thread 2 (Thread 0x7f6d5dcc1700 (LWP 32051)):
#0 0x00007f6d63ed632a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f6d62d94e73 in kudu::Subprocess::DoWait (this=0x7f6d5dcc02a0, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:817
#2 0x00007f6d62d92433 in kudu::Subprocess::Wait (this=0x7f6d5dcc02a0, wait_status=0x0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:544
#3 0x00007f6d62cf1869 in kudu::PstackWatcher::RunStackDump (argv=std::vector of length 12, capacity 16 = {...}) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:258
#4 0x00007f6d62cf1418 in kudu::PstackWatcher::RunGdbStackDump (pid=32048, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:218
#5 0x00007f6d62cf0d1a in kudu::PstackWatcher::DumpPidStacks (pid=32048, flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:169
#6 0x00007f6d62cf0cb5 in kudu::PstackWatcher::DumpStacks (flags=0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/pstack_watcher.cc:162
#7 0x0000000000513498 in kudu::<lambda()>::operator()(void) const (__closure=0x3a987bc8) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:60
#8 0x0000000000513bae in std::__invoke_impl<void, kudu::CreateAndStartTimeoutThread()::<lambda()> >(std::__invoke_other, kudu::<lambda()> &&) (__f=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:60
#9 0x0000000000513960 in std::__invoke<kudu::CreateAndStartTimeoutThread()::<lambda()> >(kudu::<lambda()> &&) (__fn=...) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:95
#10 0x0000000000514180 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x3a987bc8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
#11 0x000000000051412a in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > >::operator()(void) (this=0x3a987bc8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
#12 0x00000000005140f4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::<lambda()> > > >::_M_run(void) (this=0x3a987bc0) at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
#13 0x00007f6d67be4a4f in execute_native_thread_routine () from /tmp/dist-test-taskHr75Lv/build/debug/bin/../../../thirdparty/installed/uninstrumented/lib/librocksdb.so.7
#14 0x00007f6d63ecb6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007f6d5f2f061f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f6d5e51f240 (LWP 32048)):
#0 0x00007f6d63ed1ad3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
W20260812 09:56:22.164525 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#1 0x00007f6d62c76ade in kudu::ConditionVariable::Wait (this=0x3af33950) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/condition_variable.cc:55
W20260812 09:56:22.374850 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1786: this message will repeat every 5th retry.
I20260812 09:56:22.607968 5867 raft_consensus.cc:482] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51bce1a9cf8949ae905ebcb8ddce105b)
I20260812 09:56:22.608199 5867 raft_consensus.cc:504] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bce1a9cf8949ae905ebcb8ddce105b" member_type: VOTER last_known_addr { host: "127.31.76.1" port: 42099 } } peers { permanent_uuid: "08702823bf7945c699d002263093cd05" member_type: VOTER last_known_addr { host: "127.31.76.2" port: 43897 } } peers { permanent_uuid: "fe00076da7784496bd55b00c1661bf13" member_type: VOTER last_known_addr { host: "127.31.76.3" port: 37683 } }
I20260812 09:56:22.608783 5867 leader_election.cc:288] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099), fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683)
W20260812 09:56:22.609413 760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20260812 09:56:22.610113 760 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111)
W20260812 09:56:22.610253 764 leader_election.cc:334] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111)
I20260812 09:56:22.610322 764 leader_election.cc:302] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08702823bf7945c699d002263093cd05; no voters: 51bce1a9cf8949ae905ebcb8ddce105b, fe00076da7784496bd55b00c1661bf13
I20260812 09:56:22.610507 5867 raft_consensus.cc:2726] T 0e7e5067ea9743dd9942e27f4ea6c391 P 08702823bf7945c699d002263093cd05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
#2 0x00007f6d6bffc114 in kudu::CountDownLatch::Wait (this=0x3af33918) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/countdown_latch.h:79
W20260812 09:56:22.856195 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
#3 0x00007f6d6bb1a52a in kudu::Synchronizer::Wait (this=0x7ffeac20cb20) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/async_util.h:58
#4 0x00007f6d69be7d7d in kudu::client::KuduClient::Data::ConnectToCluster (this=0x3ab6bc20, client=0x3ac3b140, deadline=..., creds_policy=kudu::rpc::CredentialsPolicy::ANY_CREDENTIALS) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client-internal.cc:815
W20260812 09:56:24.487632 764 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1796: this message will repeat every 5th retry.
#5 0x00007f6d69bad0e4 in kudu::client::KuduClientBuilder::Build (this=0x7ffeac20ced8, client=0x3a95e6b0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/client.cc:469
W20260812 09:56:24.791419 760 consensus_peers.cc:487] T 1dd29c709b2048d3a1eee19e376c8761 P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1801: this message will repeat every 5th retry.
W20260812 09:56:24.874846 764 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer 51bce1a9cf8949ae905ebcb8ddce105b (127.31.76.1:42099): Couldn't send request to peer 51bce1a9cf8949ae905ebcb8ddce105b. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.1:42099: connect: Connection refused (error 111). This is attempt 1791: this message will repeat every 5th retry.
W20260812 09:56:24.996685 760 consensus_peers.cc:487] T 36288ce10b364d2dae07ac4e3706e40e P 08702823bf7945c699d002263093cd05 -> Peer fe00076da7784496bd55b00c1661bf13 (127.31.76.3:37683): Couldn't send request to peer fe00076da7784496bd55b00c1661bf13. Status: Network error: Client connection negotiation failed: client connection to 127.31.76.3:37683: connect: Connection refused (error 111). This is attempt 1806: this message will repeat every 5th retry.
#6 0x00007f6d69eccbd1 in kudu::tools::CreateKuduClient (master_addresses=std::vector of length 1, capacity 1 = {...}, client=0x3a95e6b0, can_see_all_replicas=true) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/tool_action_common.cc:738
#7 0x00007f6d6b844c9d in kudu::tools::RemoteKsckCluster::Connect (this=0x3a95e5a0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck_remote.cc:530
#8 0x00007f6d6b80233f in kudu::tools::Ksck::CheckClusterRunning (this=0x3ac1ac10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:460
#9 0x00007f6d6b80380a in kudu::tools::Ksck::Run (this=0x3ac1ac10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:595
#10 0x00007f6d6b805a86 in kudu::tools::Ksck::RunAndPrintResults (this=0x3ac1ac10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/ksck.cc:801
#11 0x00007f6d6bfce640 in kudu::ClusterVerifier::RunKsck (this=0x7ffeac20d6b0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:113
#12 0x00007f6d6bfce0a5 in kudu::ClusterVerifier::CheckCluster (this=0x7ffeac20d6b0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:78
#13 0x00000000004b376f in kudu::client::AllTypesItest<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> > >::RunTest(std::function<kudu::Status (kudu::client::KuduScanner*)> const&, std::function<void (kudu::client::KuduScanBatch const&, int, int*)> const&) (this=0x3aa7c900, scanner_setup=..., verifier=...) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:622
W20260812 09:56:26.546641 923 heartbeater.cc:644] Failed to heartbeat to 127.31.76.62:44395 (783 consecutive failures): Network error: Failed to ping master at 127.31.76.62:44395: Client connection negotiation failed: client connection to 127.31.76.62:44395: connect: Connection refused (error 111)
#14 0x0000000000496612 in kudu::client::AllTypesItest_TestTimestampPadding_Test<kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> > >::TestBody (this=0x3aa7c900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/all_types-itest.cc:804
#15 0x00007f6d6302ebfe in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f6d63034a28 "the test body", method=<optimized out>, object=0x3aa7c900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#16 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x3aa7c900, method=<optimized out>, location=location@entry=0x7f6d63034a28 "the test body") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#17 0x00007f6d630235db in testing::Test::Run (this=0x3aa7c900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#18 testing::Test::Run (this=0x3aa7c900) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2664
#19 0x00007f6d63023795 in testing::TestInfo::Run (this=0x3a9ae5b0) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#20 0x00007f6d63023cdf in testing::TestSuite::Run (this=0x3aa51d10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#21 testing::TestSuite::Run (this=0x3aa51d10) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2985
#22 0x00007f6d6302429f in testing::internal::UnitTestImpl::RunAllTests() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:600
#23 0x00007f6d6302f13e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f6d630365a8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x3a99a280) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest-internal-inl.h:831
#24 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x3a99a280, method=<optimized out>, location=location@entry=0x7f6d630365a8 "auxiliary test code (environments or event listeners)") at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#25 0x00007f6d6302383d in testing::UnitTest::Run() () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:1246
#26 0x00000000005144b0 in RUN_ALL_TESTS () at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#27 0x00000000005138da in main (argc=1, argv=0x7ffeac20dc18) at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20260812 09:56:26.748699 32051 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1786528586 (unix time) try "date -d @1786528586" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007d30) received by PID 32048 (TID 0x7f6d5dcc1700) from PID 32048; stack trace: ***
@ 0x7f6d61758a46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
@ 0x7f6d63ed6980 (unknown) at ??:0
@ 0x7f6d5f20de87 gsignal at ??:0
@ 0x7f6d5f20f7f1 abort at ??:0
@ 0x7f6d6174a9ad google::LogMessage::Fail() at ??:0
@ 0x7f6d6174d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6d6174a4de google::LogMessage::Flush() at ??:0
@ 0x7f6d6174ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x5135cc _ZZN4kuduL27CreateAndStartTimeoutThreadEvENKUlvE_clEv at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:63 (discriminator 9)
@ 0x513bae _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:60
@ 0x513960 _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/invoke.h:96
@ 0x514180 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:244
@ 0x51412a _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEclEv at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:253
@ 0x5140f4 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvEUlvE_EEEEE6_M_runEv at /opt/rh/devtoolset-8/root/usr/include/c++/8/thread:196
@ 0x7f6d67be4a4f execute_native_thread_routine at ??:0
@ 0x7f6d63ecb6db start_thread at ??:0
@ 0x7f6d5f2f061f clone at ??:0